Class: LMDB::Environment

Inherits:

Object

• Object
• LMDB::Environment

Defined in:

ext/lmdb_ext/lmdb_ext.c,
ext/lmdb_ext/lmdb_ext.c

Overview

The Environment is the root object for all LMDB operations.

An LMDB “environment” is a collection of one or more “databases” (key-value tables), along with transactions to modify those databases and cursors to iterate through them.

An environment – and its collection of databases – is normally stored in a directory. That directory will contain two files:

• data.mdb: all the records in all the databases in the environment

• lock.mdb: state of transactions that may be going on in the environment.

An environment can contain multiple databases. Each of the databases has a string name (“mydatabase”, “db.3.1982”). You use the database name to open the database within the environment.

Examples:

The normal pattern for using LMDB in Ruby

env = LMDB.new "databasedir"
db = env.database "databasename"
# ... do things to the database ...
env.close

Class Method Summary

• .new(path, opts) {|env| ... } ⇒ Environment

  Open an LMDB database environment.

Instance Method Summary

• #active_txn ⇒ Transaction
• #clear_flags(flags) ⇒ Object

  Clear one or more flags in the environment.

• #close ⇒ Object

  Close an environment, completing all IOs and cleaning up database state if needed.

• #copy(path) ⇒ Object

  Create a copy (snapshot) of an environment.

• #database(name, options) ⇒ Database

  Opens a database within the environment.

• #databases ⇒ Array

  Returns a list of the names of all databases in the environment, or an empty list if there are no named databases but only the defaut database.

• #flags ⇒ Array

  Return the flags that are set in this environment.

• #info ⇒ Hash

  Return useful information about an environment.

• #mapsize=(size) ⇒ Object
• #path ⇒ String

  Return the path to the database environment files.

• #set_flags(flags) ⇒ Object

  Set one or more flags in the environment.

• #stat ⇒ Hash

  Return useful statistics about an environment.

• #sync(force) ⇒ Object

  Flush the data buffers to disk.

• #transaction(readonly) {|txn| ... } ⇒ Object

  Begin a transaction.

Class Method Details

.new(path, opts) {|env| ... } ⇒ Environment

Open an LMDB database environment. The database environment is the root object for all operations on a collection of databases. It has to be opened first, before individual databases can be opened or created in the environment. The database should be closed when it is no longer needed.

The options hash on this method includes all the flags listed in #flags as well as the options documented here.

Examples:

Open environment and pass options

env = LMDB.new "dbdir", :maxdbs => 30, :mapasync => true, :writemap => true

Pass environment to block

LMDB.new "dbdir" do |env|
  # ...
end

Parameters:

• path (String) —

  the path to the files containing the database

• opts (Hash) —

  options for the database environment

Options Hash (opts):

• :mode (Number) —

  The Posix permissions to set on created files.

• :maxreaders (Number) —

  The maximum number of concurrent threads that can be executing transactions at once. Default is 126.

• :maxdbs (Number) —

  The maximum number of named databases in the environment. Not needed if only one database is being used.

• :mapsize (Number) —

  The size of the memory map to be allocated for this environment, in bytes. The memory map size is the maximum total size of the database. The size should be a multiple of the OS page size. The default size is about 10MiB.

Yields:

• (env) —

  The block to be executed with the environment. The environment is closed afterwards.

Yield Parameters:

• env (Environment) —

  The environment

Returns:

• (Environment)

See Also:

• #close
• #flags

# File 'ext/lmdb_ext/lmdb_ext.c', line 656

static VALUE environment_new(int argc, VALUE *argv, VALUE klass) {
    VALUE path, option_hash;

#ifdef RB_SCAN_ARGS_KEYWORDS
    rb_scan_args_kw(RB_SCAN_ARGS_LAST_HASH_KEYWORDS,
                    argc, argv, "1:", &path, &option_hash);
#else
    rb_scan_args(argc, argv, "1:", &path, &option_hash);
#endif

    EnvironmentOptions options = {
        .flags = MDB_NOTLS,
        .maxreaders = -1,
        .maxdbs = 128,
        .mapsize = 0,
        .mode = 0755,
    };
    if (!NIL_P(option_hash))
        rb_hash_foreach(option_hash, (int (*)(ANYARGS))environment_options,
                        (VALUE)&options);

    MDB_env* env;
    check(mdb_env_create(&env));

    Environment* environment;
    VALUE venv = Data_Make_Struct(cEnvironment, Environment, environment_mark,
                                  environment_free, environment);
    environment->env = env;
    environment->thread_txn_hash = rb_hash_new();
    environment->txn_thread_hash = rb_hash_new();

    if (options.maxreaders > 0)
        check(mdb_env_set_maxreaders(env, options.maxreaders));
    if (options.mapsize > 0)
        check(mdb_env_set_mapsize(env, options.mapsize));

    check(mdb_env_set_maxdbs(env, options.maxdbs <= 0 ? 1 : options.maxdbs));
    VALUE expanded_path = rb_file_expand_path(path, Qnil);
    check(mdb_env_open(env, StringValueCStr(expanded_path), options.flags,
                       options.mode));

    if (rb_block_given_p())
        return rb_ensure(rb_yield, venv, environment_close, venv);

    return venv;
}

Instance Method Details

#active_txn ⇒ Transaction

Returns the current active transaction on this thread in the environment.

Examples:

env.transaction do |t|
  active = env.active_txn
  # active should equal t
end

Returns:

• (Transaction) —

  the current active transaction on this thread in the environment.

# File 'ext/lmdb_ext/lmdb_ext.c', line 807

static VALUE environment_active_txn(VALUE self) {
        ENVIRONMENT(self, environment);
        return rb_hash_aref(environment->thread_txn_hash, rb_thread_current());
}

#clear_flags(flags) ⇒ Object

Clear one or more flags in the environment. The available flags are defined in #flags.

Examples:

env.clear_flags :nosync, :writemap

Parameters:

• flags (Array) —

  Array of flag names (symbols) to clear

Returns:

• nil

Raises:

• (Error) —

  if an invalid flag name is specified

See Also:

• #flags

# File 'ext/lmdb_ext/lmdb_ext.c', line 793

static VALUE environment_clear_flags(int argc, VALUE* argv, VALUE self) {
        environment_change_flags(argc, argv, self, 0);
        return Qnil;
}

#close ⇒ Object

Close an environment, completing all IOs and cleaning up database state if needed.

Examples:

env = LMDB.new('abc')
# ...various operations on the environment...
env.close

# File 'ext/lmdb_ext/lmdb_ext.c', line 475

static VALUE environment_close(VALUE self) {
        ENVIRONMENT(self, environment);
        mdb_env_close(environment->env);
        environment->env = 0;
        return Qnil;
}

#copy(path) ⇒ Object

Create a copy (snapshot) of an environment. The copy can be used as a backup. The copy internally uses a read-only transaction to ensure that the copied data is serialized with respect to database updates.

Parameters:

• path (String) —

  The directory in which the copy will reside. This directory must already exist and be writable but must otherwise be empty.

Returns:

• nil

Raises:

• (Error) —

  when there is an error creating the copy.

# File 'ext/lmdb_ext/lmdb_ext.c', line 568

static VALUE environment_copy(VALUE self, VALUE path) {
        ENVIRONMENT(self, environment);
        VALUE expanded_path = rb_file_expand_path(path, Qnil);
        check(mdb_env_copy(environment->env, StringValueCStr(expanded_path)));
        return Qnil;
}

#database(name, options) ⇒ Database

Opens a database within the environment.

Note that a database is opened or created within a transaction. If the open creates a new database, the database is not available for other operations in other transactions until the transaction that is creating the database commits. If the transaction creating the database aborts, the database is not created.

Parameters:

• name (String) —

  Optional name for the database to be opened.

• options (Hash) —

  Options for the database.

Options Hash (options):

• :reversekey (Boolean) —

  Keys are strings to be compared in reverse order, from the end of the strings to the beginning. By default, Keys are treated as strings and compared from beginning to end.

• :dupsort (Boolean) —

  Duplicate keys may be used in the database. (Or, from another perspective, keys may have multiple data items, stored in sorted order.) By default keys must be unique and may have only a single data item.

• :integerkey (Boolean) —

  Keys are binary integers in native byte order.

• :dupfixed (Boolean) —

  This flag may only be used in combination with :dupsort. This option tells the library that the data items for this database are all the same size, which allows further optimizations in storage and retrieval.

• :integerdup (Boolean) —

  This option specifies that duplicate data items are also integers, and should be sorted as such.

• :reversedup (Boolean) —

  This option specifies that duplicate data items should be compared as strings in reverse order.

• :create (Boolean) —

  Create the named database if it doesn’t exist. This option is not allowed in a read-only transaction or a read-only environment.

Returns:

• (Database) —

  newly-opened database

Raises:

• (Error) —

  if there is an error opening the database

# File 'ext/lmdb_ext/lmdb_ext.c', line 942

static VALUE environment_database(int argc, VALUE *argv, VALUE self) {
    ENVIRONMENT(self, environment);
    if (!active_txn(self))
        return call_with_transaction(self, self, "database", argc, argv, 0);

    VALUE name, option_hash;
#ifdef RB_SCAN_ARGS_KEYWORDS
    rb_scan_args_kw(RB_SCAN_ARGS_KEYWORDS,
                    argc, argv, "01:", &name, &option_hash);
#else
    rb_scan_args(argc, argv, "01:", &name, &option_hash);
#endif


    int flags = 0;
    if (!NIL_P(option_hash))
        rb_hash_foreach(option_hash, (int (*)(ANYARGS))database_flags,
                        (VALUE)&flags);

    MDB_dbi dbi;
    check(mdb_dbi_open(need_txn(self), NIL_P(name) ? 0 : StringValueCStr(name),
                       flags, &dbi));

    Database* database;
    VALUE vdb = Data_Make_Struct(cDatabase, Database, database_mark, free,
                                 database);
    database->dbi = dbi;
    database->env = self;

    return vdb;
}

#databases ⇒ Array

Returns The names of all databases in the environment.

Returns a list of the names of all databases in the environment, or an empty list if there are no named databases but only the defaut database.

Returns:

• (Array) —

  The names of all databases in the environment.

Raises:

• (Error) —

  If there is an error opening the main database.

# File 'ext/lmdb_ext/lmdb_ext.c', line 982

static VALUE environment_databases(VALUE self) {
    ENVIRONMENT(self, environment);
    if (!active_txn(self))
        return call_with_transaction(self, self, "databases", 0, 0, MDB_RDONLY);

    MDB_dbi dbi;
    MDB_cursor *cursor;
    MDB_txn *txn = need_txn(self);
    MDB_val key;

    check(mdb_dbi_open(txn, NULL, 0, &dbi));
    check(mdb_cursor_open(txn, dbi, &cursor));

    VALUE ret = rb_ary_new();
    while (mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP) == MDB_SUCCESS) {
        char *intern_db_name;
        MDB_dbi db;
        VALUE db_name;

        if (memchr(key.mv_data, '\0', key.mv_size))
            continue;

        intern_db_name = malloc(key.mv_size + 1);
        memcpy(intern_db_name, key.mv_data, key.mv_size);
        intern_db_name[key.mv_size] = '\0';

        if (mdb_dbi_open(txn, intern_db_name, 0, &db) == MDB_SUCCESS) {
            mdb_dbi_close(environment->env, db);
            db_name = rb_str_new(key.mv_data, key.mv_size);
            rb_ary_push(ret, db_name);
        }
        free(intern_db_name);
    }

    mdb_cursor_close(cursor);

    return ret;
}

#flags ⇒ Array

Return the flags that are set in this environment. The environment flags are:

• :fixedmap Use a fixed address for the mmap region.

• :nosubdir By default, MDB creates its environment in a directory whose pathname is given in path, and creates its data and lock files under that directory. With this option, path is used as-is for the database main data file. The database lock file is the path with “-lock” appended.

• :nosync Don’t flush system buffers to disk when committing a transaction. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often #sync is called. However, if the filesystem preserves write order and the :writemap flag is not used, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). That is, database integrity is maintained, but a system crash may undo the final transactions. Note that :nosync :writemap+ leaves the system with no hint for when to write transactions to disk, unless #sync is called. :mapasync :writemap+ may be preferable.

• :rdonly Open the environment in read-only mode. No write operations will be allowed. MDB will still modify the lock file - except on read-only filesystems, where MDB does not use locks.

• :nometasync Flush system buffers to disk only once per transaction, omit the metadata flush. Defer that until the system flushes files to disk, or next non-MDB_RDONLY commit or #sync. This optimization maintains database integrity, but a system crash may undo the last committed transaction. That is, it preserves the ACI (atomicity, consistency, isolation) but not D (durability) database property.

• :writemap Use a writeable memory map unless :rdonly is set. This is faster and uses fewer mallocs, but loses protection from application bugs like wild pointer writes and other bad updates into the database. Incompatible with nested transactions.

• :mapasync When using :writemap, use asynchronous flushes to disk. As with :nosync, a system crash can then corrupt the database or lose the last transactions. Calling #sync ensures on-disk database integrity until next commit.

• :notls Don’t use thread-local storage.

Examples:

env = LMDB.new "abc", :writemap => true, :nometasync => true
env.flags           #=> [:writemap, :nometasync]

Returns:

• (Array) —

  Array of flag symbols

# File 'ext/lmdb_ext/lmdb_ext.c', line 720

static VALUE environment_flags(VALUE self) {
        unsigned int flags;
        ENVIRONMENT(self, environment);
        check(mdb_env_get_flags(environment->env, &flags));

        VALUE ret = rb_ary_new();
#define FLAG(const, name) if (flags & MDB_##const) rb_ary_push(ret, ID2SYM(rb_intern(#name)));
#include "env_flags.h"
#undef FLAG

        return ret;
}

#info ⇒ Hash

Return useful information about an environment.

• :mapaddr The memory address at which the database is mapped, if fixed

• :mapsize The size of the data memory map

• :last_pgno ID of the last used page

• :last_txnid ID of the last committed transaction

• :maxreaders Max reader slots in the environment

• :numreaders Max readers slots in the environment

Returns:

• (Hash)

# File 'ext/lmdb_ext/lmdb_ext.c', line 536

static VALUE environment_info(VALUE self) {
        MDB_envinfo info;

        ENVIRONMENT(self, environment);
        check(mdb_env_info(environment->env, &info));

        VALUE ret = rb_hash_new();

#define INFO_SET(name) rb_hash_aset(ret, ID2SYM(rb_intern(#name)), SIZET2NUM((size_t)info.me_##name));
        INFO_SET(mapaddr);
        INFO_SET(mapsize);
        INFO_SET(last_pgno);
        INFO_SET(last_txnid);
        INFO_SET(maxreaders);
        INFO_SET(numreaders);
#undef INFO_SET

        return ret;
}

#mapsize=(size) ⇒ Object

# File 'ext/lmdb_ext/lmdb_ext.c', line 745

static VALUE environment_set_mapsize(VALUE self, VALUE size) {
        ENVIRONMENT(self, environment);
        check(mdb_env_set_mapsize(environment->env, NUM2LONG(size)));
        return Qnil;
}

#path ⇒ String

Return the path to the database environment files

Returns:

• (String) —

  the path that was used to open the environment.

# File 'ext/lmdb_ext/lmdb_ext.c', line 738

static VALUE environment_path(VALUE self) {
        const char* path;
        ENVIRONMENT(self, environment);
        check(mdb_env_get_path(environment->env, &path));
        return rb_str_new2(path);
}

#set_flags(flags) ⇒ Object

Set one or more flags in the environment. The available flags are defined in #flags.

Examples:

env.set_flags :nosync, :writemap

Parameters:

• flags (Array) —

  Array of flag names (symbols) to set

Returns:

• nil

Raises:

• (Error) —

  if an invalid flag name is specified

See Also:

• #flags

# File 'ext/lmdb_ext/lmdb_ext.c', line 778

static VALUE environment_set_flags(int argc, VALUE* argv, VALUE self) {
        environment_change_flags(argc, argv, self, 1);
        return Qnil;
}

#stat ⇒ Hash

Return useful statistics about an environment.

• :psize Size of a database page

• :depth Depth (height) of the B-tree

• :branch_pages Number of internal (non-leaf) pages

• :leaf_pages Number of leaf pages

• :overflow_pages Number of overflow pages

• :entries Number of data items

Returns:

• (Hash) —

  the statistics

# File 'ext/lmdb_ext/lmdb_ext.c', line 518

static VALUE environment_stat(VALUE self) {
        ENVIRONMENT(self, environment);
        MDB_stat stat;
        check(mdb_env_stat(environment->env, &stat));
        return stat2hash(&stat);
}

#sync(force) ⇒ Object

Flush the data buffers to disk.

Data is always written to disk when Transaction#commit is called, but the operating system may keep it buffered. MDB always flushes the OS buffers upon commit as well, unless the environment was opened with :nosync or in part :nometasync.

Parameters:

• force (Boolean) —

  If true, force a synchronous flush. Otherwise if the environment has the :nosync flag set the flushes will be omitted, and with :mapasync they will be asynchronous.

# File 'ext/lmdb_ext/lmdb_ext.c', line 588

static VALUE environment_sync(int argc, VALUE *argv, VALUE self) {
        ENVIRONMENT(self, environment);

        VALUE force;
        rb_scan_args(argc, argv, "01", &force);

        check(mdb_env_sync(environment->env, RTEST(force)));
        return Qnil;
}

#transaction(readonly) {|txn| ... } ⇒ Object

Note:

Transactions can be nested.

Note:

do a ‘break txn.commit` if you want to exit from the block early without aborting.

Begin a transaction. Takes a block to run the body of the transaction. A transaction commits when it exits the block successfully. A transaction aborts when it raises an exception or calls Transaction#abort.

Examples:

db = env.database "mydata"
env.transaction do |txn1|
  db['a'] = 1
  env.transaction do |txn2|
    # txn2 is nested in txn1
    db['a'] = 2
    db['a']                    #=> 2
    txn2.abort
  end
  db['a']                      #=> 1
  env.transaction do
    db['a'] = 3
  end
end
db['a']                        #=> 3

Parameters:

• readonly (Boolean) —

  This transaction will not perform any write operations

Yields:

• (txn) —

  The block to be executed with the body of the transaction.

Yield Parameters:

• txn (Transaction) —

  An optional transaction argument

# File 'ext/lmdb_ext/lmdb_ext.c', line 885

static VALUE environment_transaction(int argc, VALUE *argv, VALUE self) {
        rb_need_block();

        VALUE readonly;
        rb_scan_args(argc, argv, "01", &readonly);
        unsigned int flags = RTEST(readonly) ? MDB_RDONLY : 0;

        return with_transaction(self, rb_yield, Qnil, flags);
}