Redland RDF Application Framework - Storage Modules

Introduction

Redland includes several modules that implement the storage API and provide a variety of different features and functionality. This document gives the details of what each implementation module provides and the storage options used.

Store types:

Store 'hashes'

This module is always present (cannot be removed) and provides indexed storage using Redland hashes to store various combinations of subject, predicate and object for faster access. Context nodes are also stored in a hash when used. The hashes may be in-memory (always availabe) or persistent via Sleepycat/Berkeley DB (BDB) versions 2-4. It is the most mature and primary persistent store and suitable for large models, tested in the 2-3 million range..

The main option requiring setting is the hash-type which must be one of the supported Redland hashes. Hash type memory is always available and if BDB has been compiled in, bdb is also available. Option dir can be used to set the destination directory for the BDB files when used. Boolean option new can be set to force creation or truncation of a persistent hashed store. The storage name must be given for hash type bdb since it is used for a filename.

The module provides optional contexts support enabled when boolean storage option contexts is set. This can be used with any hash type.

Examples:

  /* A new BDB hashed persistent store in the current directory */
  storage=librdf_new_storage(world, "hashes", "db1",
                             "new='yes',hash-type='bdb',dir='.'");

  /* Hashed in-memory store */
  storage=librdf_new_storage(world, "hashes", NULL,
                             "hash-type='memory'");

  /* An existing BDB hashed persistent store in dir /somewhere */
  storage=librdf_new_storage(world, "hashes", "dv2", 
                             "hash-type='bdb',dir='/somewhere'");

  /* An existing BDB hashed store with contexts */
  storage=librdf_new_storage(world, "hashes", "db3", 
                             "hash-type='bdb',contexts='yes'");

In Python:

  from RDF import *
  ...
  # Create a new BDB store
  storage = HashStorage("db4", options="new='yes',hash-type='bdb'")

In Perl:

  use RDF::Redland;
  ...
  # Open an existing BDB store
  $storage=new RDF::Redland::Storage("hashes", "db5", 
                                     "hash-type='bdb',dir='.'");

Summary:

Store 'trees'

This module is always present (cannot be removed) and provides indexed storage using several balanced trees to store statements for fast querying. This store is not persistent, but is suitable for large models capable of fitting in main memory.

By default, the store is fully indexed providing good performance for all types of queries. Options can be used to select only specific indices to save memory and make insertion and deletion of statements faster. The four boolean indexing options are index-spo, index-sop, index-ops and index-pso. An index is fast for triple patterns where the variables are on the right hand side of the index ordering, e.g. the spo (subject, predicate, object) index will be fast for (s p o) (s p ?o) and (s ?p ?o) queries. The ideal index for every type of triple pattern is:

Any index:
(s p o)

(s p o):
(s p ?o), (s ?p ?o)

(o p s):
(?s p o), (?s ?p o)

(s o p):
(s ?p o)

(p s o):
(?s p ?o)

With full indexing the space used is roughly equivalent to the hashes store. Insertion and deletion with 2 indices will be roughly twice as fast as with 4 indices, etc. In the majority of cases, full indexing will be fine and there is no need to worry about selecting the right index for queries.

Examples:

  /* A fully indexed tree store */
  storage=librdf_new_storage(world, "trees", NULL, NULL);

  /* A tree store with only a (s p o) index */
  storage=librdf_new_storage(world, "trees", NULL, "index-spo='yes'");
  
  /* A tree store with only (s p o) and (o p s) indices
   * (fast for everything but queries with predicate variables */
  storage=librdf_new_storage(world, "trees", NULL,
    "index-spo='yes',index-ops='yes'");

Summary:

Store 'memory'

This module is always present (cannot be removed) and provides a simple and fast in-memory store with no persistence. It is the default store if no store name is given to the storage constructors.

The memory store is not suitable for large in-memory models since it does not do any indexing. For that, use the hash indexed store with hash-type of memory.

The module provides optional contexts support enabled when boolean storage option contexts is set.

Examples:

  /* Explictly named memory storage */
  storage=librdf_new_storage(world, "memory", NULL, NULL);

  /* Default storage type, which is memory */
  storage=librdf_new_storage(world, NULL, NULL, NULL);

  /* In-memory store with contexts */
  storage=librdf_new_storage(world, NULL, NULL, "contexts='yes'");

Summary:

Store 'file'

This module provides an in-memory model (internally, using a memory storage) initialised from the RDF/XML content in a file. The file is given as the storage name and assumed to exist on opening. When a model or storage sync method is called or the model or store is closed, a new file is created, the old file renamed to a backup and the new file renamed to replace it. This store was added in Redland 0.9.15

There are no options for this store and contexts are not supported.

Example:

  /* File based store from thing.rdf file */
  storage=librdf_new_storage(world, "file", "thing.rdf", NULL);

Summary:

Store 'mysql'

This module was written by Morten Frederiksen and is compiled in when MySQL 3 or 4 is available. This store provides storage using the MySQL open source database including contexts. It is a new store added in Redland 0.9.15 and is still under development. It has however been tested with several million triples and deployed.

There are several options required with the mysql storage in order to connect to the database. These are:

NOTE: Take care exposing the password as for example, program arguments or environment variables. The rdfproc utility can with help this by reading the password from standard input. Inside programs, one way to prevent storing the password in a string is to construct a Redland hash of the storage options such as via librdf_hash_from_string and use librdf_new_storage_with_options to create a storage. The rdfproc utility source code demonstrates this.

The storage name parameter given to the storage constructor librdf_new_storage is used inside the mysql store to allow multiple stores inside one MySQL database instance as parameterised with the above optiosn.

If boolean option new is given, any existing MySQL database named by the storage option database, say db will be dropped and the appropriate new tables created. The MySQL database db must already exist, such as made with the MySQL create database db command and the appropriate priviledges set so that the user and password work.

If boolean option reconnect is given, MySQL reconnection will be enabled so that if the database conneciton is dropped, MySQL will attempt to reconnect.

This store always provides contexts; the boolean storage option contexts is not checked.

Examples:

  /* A new MySQL store */
  storage=librdf_new_storage(world, "mysql", "db1",
      "new='yes',host='localhost',database='red',user='foo','password='bar'");

  /* A different, existing MySQL store db2 in the same database as above */
  storage=librdf_new_storage(world, "mysql", "db2",
      "host='localhost',database='red',user='foo','password='bar'");

  /* An existing MySQL store on a different database server */
  storage=librdf_new_storage(world, "mysql", "db3",
      "host='db.example.org',database='abc',user='baz','password='blah'");

  /* Opening with an options hash */
  options=librdf_new_hash(world, NULL);
  librdf_hash_from_string(options, 
      "host='db.example.org',database='abc',user='baz'");
  librdf_hash_put_strings(options, "password", user_password);
  storage=librdf_new_storage_with_options(world, "mysql", "db4", options);

In PHP:

  # An existing store
  $storage=librdf_new_storage($world, 'mysql', 'db4', 
      "host='127.0.0.1',database='xyz',user='foo',password='blah'");

Summary:

Store 'postgresql'

This module was written by Shi Wenzhong based on the MySQL store written by Morten Frederiksen and is compiled in when PostgreSQL is available. This store provides storage using the PostgreSQL open source database including contexts. This store was added in This Redland 1.0.3.

There are several options required with the postgresql storage in order to connect to the database. These are:

NOTE: Before Redland 1.0.5, the parameter dbname had to be used instead of database for the Postgresql database name

NOTE: Take care exposing the password as for example, program arguments or environment variables. The rdfproc utility can with help this by reading the password from standard input. Inside programs, one way to prevent storing the password in a string is to construct a Redland hash of the storage options such as via librdf_hash_from_string and use librdf_new_storage_with_options to create a storage. The rdfproc utility source code demonstrates this.

The storage name parameter given to the storage constructor librdf_new_storage is used inside the postgresql store to allow multiple stores inside one PostgreSQL database instance as parameterised with the above optiosn.

If boolean option new is given, any existing PostgreSQL database named by the storage option database, say db will be dropped and the appropriate new tables created. The PostgreSQL database db must already exist, such as made with the PostgreSQL create database db command and the appropriate priviledges set so that the user and password work.

This store always provides contexts; the boolean storage option contexts is not checked.

Examples:

  /* A new PostgreSQL store */
  storage=librdf_new_storage(world, "postgresql", "db1",
      "new='yes',host='localhost',database='red',user='foo','password='bar'");

  /* A different, existing PostgreSQL store db2 in the same database as above */
  storage=librdf_new_storage(world, "postgresql", "db2",
      "host='localhost',database='red',user='foo','password='bar'");

  /* An existing PostgreSQL store on a different database server */
  storage=librdf_new_storage(world, "postgresql", "db3",
      "host='db.example.org',database='abc',user='baz','password='blah'");

  /* Opening with an options hash */
  options=librdf_new_hash(world, NULL);
  librdf_hash_from_string(options, 
      "host='db.example.org',database='abc',user='baz'");
  librdf_hash_put_strings(options, "password", user_password);
  storage=librdf_new_storage_with_options(world, "postgresql", "db4", options);

In PHP:

  # An existing store
  $storage=librdf_new_storage($world, 'postgresql', 'db4', 
      "host='127.0.0.1',database='xyz',user='foo',password='blah'");

Summary:

Store 'sqlite'

This module provides storage via the SQLite relational database when available and supports SQLite V2 and V3. It was added in Redland 1.0.0 and is of beta quality. This store provides triples and contexts.

The only option respected by this store is the new one to create a new store, destroying any existing store.

Summary:

Store 'tstore'

This module provides storage via the AKT Triplestore when available. It was added in Redland 0.9.15 and is alpha quality - not complete or tested significantly (although the AKT store itself is used in production). This store provides a basic triple API but no redland contexts. The underlying RDQL support and inference is not yet exposed in Redland but may be in future.

There are several options required with the tstore storage in order to connect to the tstore database (which uses MySQL). These are:

NOTE: Take care exposing the password as for example, program arguments or environment variables. The rdfproc utility can with help this by reading the password from standard input. Inside programs, one way to prevent storing the password in a string is to construct a Redland hash of the storage options such as via librdf_hash_from_string and use librdf_new_storage_with_options to create a storage. The rdfproc utility source code demonstrates this.

Summary:

Store 'uri'

This module provides an in-memory model (internally, using a memory storage) initialised from the RDF/XML content in a URI. The URI is given as the storage name and on closing, the model is destroyed. This is a new store added in Redland 0.9.15 and is still under development. In future it may be extended to allow saving the store to the URI.

There are no options for this store and contexts are not supported.

Example:

  /* Read URI content into a store */
  storage=librdf_new_storage(world, "uri", 
     "http://example.org/content.rdf", NULL);

Summary:

Copyright 2004-2007 Dave Beckett, Copyright 2004 University of Bristol