Class: LMDB::Environment

Inherits:
Object
  • Object
show all
Defined in:
ext/lmdb_ext/lmdb_ext.c,
ext/lmdb_ext/lmdb_ext.c

Overview

The Environment is the root object for all LMDB operations.

An LMDB “environment” is a collection of one or more “databases” (key-value tables), along with transactions to modify those databases and cursors to iterate through them.

An environment – and its collection of databases – is normally stored in a directory. That directory will contain two files:

  • data.mdb: all the records in all the databases in the environment

  • lock.mdb: state of transactions that may be going on in the environment.

An environment can contain multiple databases. Each of the databases has a string name (“mydatabase”, “db.3.1982”). You use the database name to open the database within the environment.

Examples:

The normal pattern for using LMDB in Ruby

env = LMDB.new "databasedir"
db = env.database "databasename"
# ... do things to the database ...
env.close

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.new(path, opts) {|env| ... } ⇒ Environment

Open an LMDB database environment. The database environment is the root object for all operations on a collection of databases. It has to be opened first, before individual databases can be opened or created in the environment. The database should be closed when it is no longer needed.

The options hash on this method includes all the flags listed in #flags as well as the options documented here.

Examples:

Open environment and pass options

env = LMDB.new "dbdir", :maxdbs => 30, :mapasync => true, :writemap => true

Pass environment to block

LMDB.new "dbdir" do |env|
  # ...
end

Parameters:

  • path (String) —

    the path to the files containing the database

  • opts (Hash) —

    options for the database environment

Options Hash (opts):

  • :mode (Number) —

    The Posix permissions to set on created files.

  • :maxreaders (Number) —

    The maximum number of concurrent threads that can be executing transactions at once. Default is 126.

  • :maxdbs (Number) —

    The maximum number of named databases in the environment. Not needed if only one database is being used.

  • :mapsize (Number) —

    The size of the memory map to be allocated for this environment, in bytes. The memory map size is the maximum total size of the database. The size should be a multiple of the OS page size. The default size is about 10MiB.

Yields:

  • (env) —

    The block to be executed with the environment. The environment is closed afterwards.

Yield Parameters:

Returns:

See Also:



965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 965

static VALUE environment_new(int argc, VALUE *argv, VALUE klass) {
  VALUE path, option_hash;

#ifdef RB_SCAN_ARGS_KEYWORDS
  rb_scan_args_kw(RB_SCAN_ARGS_LAST_HASH_KEYWORDS,
                  argc, argv, "1:", &path, &option_hash);
#else
  rb_scan_args(argc, argv, "1:", &path, &option_hash);
#endif

  EnvironmentOptions options = {
    // .flags = MDB_NOTLS,
    .flags = 0,
    .maxreaders = -1,
    .maxdbs = 128,
    .mapsize = 0,
    .mode = 0755,
  };
  if (!NIL_P(option_hash))
    rb_hash_foreach(option_hash, (int (*)(ANYARGS))environment_options,
                    (VALUE)&options);

  MDB_env* env;
  check(mdb_env_create(&env));

  Environment* environment;
  VALUE venv = TypedData_Make_Struct(cEnvironment, Environment,
                                     &lmdb_environment_type, environment);
  environment->env = env;
  environment->thread_txn_hash = rb_hash_new();
  environment->txn_thread_hash = rb_hash_new();
  environment->rw_txn_thread   = Qnil;

  if (options.maxreaders > 0)
    check(mdb_env_set_maxreaders(env, options.maxreaders));
  if (options.mapsize > 0)
    check(mdb_env_set_mapsize(env, options.mapsize));

  check(mdb_env_set_maxdbs(env, options.maxdbs <= 0 ? 1 : options.maxdbs));
  VALUE expanded_path = rb_file_expand_path(path, Qnil);
  check(mdb_env_open(env, StringValueCStr(expanded_path), options.flags,
                     options.mode));

  if (rb_block_given_p())
    return rb_ensure(rb_yield, venv, environment_close, venv);

  return venv;
}

Instance Method Details

#[] ⇒ Database

Returns a database handle.

Accesses an existing database. Will raise if it doesn’t exist.

Parameters:

  • name (String, Symbol, nil) —

    name for the database to be opened.

Returns:

Raises:

  • (Error) —

    if there is an error opening the database



1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1367

static VALUE environment_database_aref(VALUE self, VALUE name) {
  // If they pass nil explicitly, treat it as zero arguments (anonymous database)
  int argc = NIL_P(name) ? 0 : 1;

  // Package it up into an argv array
  VALUE argv[1];
  argv[0] = name;

  // Call your existing function directly
  return environment_database(argc, argv, self);
}

#active_txn ⇒ Transaction

Returns the current active transaction on this thread in the environment.

Examples:

env.transaction do |t|
  active = env.active_txn
  # active should equal t
end

Returns:

  • (Transaction) —

    the current active transaction on this thread in the environment.



1118
1119
1120
1121
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1118

static VALUE environment_active_txn(VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  return rb_hash_aref(environment->thread_txn_hash, rb_thread_current());
}

#clear_flags(flags) ⇒ Object

Clear one or more flags in the environment. The available flags are defined in #flags.

Examples:

env.clear_flags :nosync, :writemap

Parameters:

  • flags (Array) —

    Array of flag names (symbols) to clear

Returns:

  • nil

Raises:

  • (Error) —

    if an invalid flag name is specified

See Also:



1104
1105
1106
1107
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1104

static VALUE environment_clear_flags(int argc, VALUE* argv, VALUE self) {
  environment_change_flags(argc, argv, self, 0);
  return Qnil;
}

#close ⇒ Object

Close an environment, completing all IOs and cleaning up database state if needed.

Examples:

env = LMDB.new('abc')
# ...various operations on the environment...
env.close


758
759
760
761
762
763
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 758

static VALUE environment_close(VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  mdb_env_close(environment->env);
  environment->env = 0;
  return Qnil;
}

#copy(path) ⇒ Object

Create a copy (snapshot) of an environment. The copy can be used as a backup. The copy internally uses a read-only transaction to ensure that the copied data is serialized with respect to database updates.

Parameters:

  • path (String) —

    The directory in which the copy will reside. This directory must already exist and be writable but must otherwise be empty.

Returns:

  • nil

Raises:

  • (Error) —

    when there is an error creating the copy.



877
878
879
880
881
882
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 877

static VALUE environment_copy(VALUE self, VALUE path) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  VALUE expanded_path = rb_file_expand_path(path, Qnil);
  check(mdb_env_copy(environment->env, StringValueCStr(expanded_path)));
  return Qnil;
}

#database(name, options) ⇒ Database

Opens a database within the environment.

Note that a database is opened or created within a transaction. If the open creates a new database, the database is not available for other operations in other transactions until the transaction that is creating the database commits. If the transaction creating the database aborts, the database is not created.

Parameters:

  • name (String, Symbol, nil) —

    Optional name for the database to be opened.

  • options (Hash) —

    Options for the database.

Options Hash (options):

  • :reversekey (Boolean) —

    Keys are strings to be compared in reverse order, from the end of the strings to the beginning. By default, Keys are treated as strings and compared from beginning to end.

  • :dupsort (Boolean) —

    Duplicate keys may be used in the database. (Or, from another perspective, keys may have multiple data items, stored in sorted order.) By default keys must be unique and may have only a single data item.

  • :integerkey (Boolean) —

    Keys are binary integers in native byte order.

  • :dupfixed (Boolean) —

    This flag may only be used in combination with :dupsort. This option tells the library that the data items for this database are all the same size, which allows further optimizations in storage and retrieval.

  • :integerdup (Boolean) —

    This option specifies that duplicate data items are also integers, and should be sorted as such.

  • :reversedup (Boolean) —

    This option specifies that duplicate data items should be compared as strings in reverse order.

  • :create (Boolean) —

    Create the named database if it doesn’t exist. This option is not allowed in a read-only transaction or a read-only environment.

Returns:

Raises:

  • (Error) —

    if there is an error opening the database



1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1318

static VALUE environment_database(int argc, VALUE *argv, VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);

  if (!active_txn(self))
    return call_with_transaction(self, self, "database", argc, argv, 0);

  VALUE name        = Qnil;
  VALUE option_hash = Qnil;

  rb_scan_args_kw(RB_SCAN_ARGS_LAST_HASH_KEYWORDS,
                  argc, argv, "01:", &name, &option_hash);

  int flags = 0;
  if (!NIL_P(option_hash))
    rb_hash_foreach(option_hash, (int (*)(ANYARGS))database_flags,
                    (VALUE)&flags);

  MDB_dbi dbi;

  if (name && !NIL_P(name)) {
    if (RB_TYPE_P(name, T_SYMBOL))
      name = rb_sym2str(name);

    Check_Type(name, T_STRING);
  }

  check(mdb_dbi_open(need_txn(self), NIL_P(name) ? 0 : StringValueCStr(name),
                     flags, &dbi));

  Database* database;
  VALUE vdb = TypedData_Make_Struct(cDatabase, Database,
                                    &lmdb_database_type, database);
  database->dbi = dbi;
  database->env = self;

  return vdb;
}

#databases ⇒ Array

Returns The names of all databases in the environment.

Returns a list of the names of all databases in the environment, or an empty list if there are no named databases but only the defaut database.

Returns:

  • (Array) —

    The names of all databases in the environment.

Raises:

  • (Error) —

    If there is an error opening the main database.



1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1387

static VALUE environment_databases(VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  if (!active_txn(self))
    return call_with_transaction(self, self, "databases", 0, 0, MDB_RDONLY);

  MDB_dbi dbi;
  MDB_cursor *cursor;
  MDB_txn *txn = need_txn(self);
  MDB_val key;

  check(mdb_dbi_open(txn, NULL, 0, &dbi));
  check(mdb_cursor_open(txn, dbi, &cursor));

  VALUE ret = rb_ary_new();
  while (mdb_cursor_get(cursor, &key, NULL, MDB_NEXT_NODUP) == MDB_SUCCESS) {
    char *intern_db_name;
    MDB_dbi db;
    VALUE db_name;

    if (memchr(key.mv_data, '\0', key.mv_size))
      continue;

    intern_db_name = malloc(key.mv_size + 1);
    memcpy(intern_db_name, key.mv_data, key.mv_size);
    intern_db_name[key.mv_size] = '\0';

    if (mdb_dbi_open(txn, intern_db_name, 0, &db) == MDB_SUCCESS) {
      mdb_dbi_close(environment->env, db);
      db_name = rb_str_new(key.mv_data, key.mv_size);
      rb_ary_push(ret, db_name);
    }
    xfree(intern_db_name);
  }

  mdb_cursor_close(cursor);

  return ret;
}

#flags ⇒ Array

Return the flags that are set in this environment. The environment flags are:

  • :fixedmap Use a fixed address for the mmap region.

  • :nosubdir By default, MDB creates its environment in a directory whose pathname is given in path, and creates its data and lock files under that directory. With this option, path is used as-is for the database main data file. The database lock file is the path with “-lock” appended.

  • :nosync Don’t flush system buffers to disk when committing a transaction. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often #sync is called. However, if the filesystem preserves write order and the :writemap flag is not used, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). That is, database integrity is maintained, but a system crash may undo the final transactions. Note that :nosync :writemap+ leaves the system with no hint for when to write transactions to disk, unless #sync is called. :mapasync :writemap+ may be preferable.

  • :rdonly Open the environment in read-only mode. No write operations will be allowed. MDB will still modify the lock file - except on read-only filesystems, where MDB does not use locks.

  • :nometasync Flush system buffers to disk only once per transaction, omit the metadata flush. Defer that until the system flushes files to disk, or next non-MDB_RDONLY commit or #sync. This optimization maintains database integrity, but a system crash may undo the last committed transaction. That is, it preserves the ACI (atomicity, consistency, isolation) but not D (durability) database property.

  • :writemap Use a writeable memory map unless :rdonly is set. This is faster and uses fewer mallocs, but loses protection from application bugs like wild pointer writes and other bad updates into the database. Incompatible with nested transactions.

  • :mapasync When using :writemap, use asynchronous flushes to disk. As with :nosync, a system crash can then corrupt the database or lose the last transactions. Calling #sync ensures on-disk database integrity until next commit.

  • :notls Don’t use thread-local storage.

Examples:

env = LMDB.new "abc", :writemap => true, :nometasync => true
env.flags           #=> [:writemap, :nometasync]

Returns:

  • (Array) —

    Array of flag symbols



1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1031

static VALUE environment_flags(VALUE self) {
  unsigned int flags;
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  check(mdb_env_get_flags(environment->env, &flags));

  VALUE ret = rb_ary_new();
#define FLAG(const, name) if (flags & MDB_##const) rb_ary_push(ret, ID2SYM(rb_intern(#name)));
#include "env_flags.h"
#undef FLAG

  return ret;
}

#info ⇒ Hash

Return useful information about an environment.

  • :mapaddr The memory address at which the database is mapped, if fixed

  • :mapsize The size of the data memory map

  • :last_pgno ID of the last used page

  • :last_txnid ID of the last committed transaction

  • :maxreaders Max reader slots in the environment

  • :numreaders Max readers slots in the environment

Returns:

  • (Hash) 


845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 845

static VALUE environment_info(VALUE self) {
  MDB_envinfo info;

  ENVIRONMENT(self, &lmdb_environment_type, environment);
  check(mdb_env_info(environment->env, &info));

  VALUE ret = rb_hash_new();

#define INFO_SET(name) rb_hash_aset(ret, ID2SYM(rb_intern(#name)), SIZET2NUM((size_t)info.me_##name));
  INFO_SET(mapaddr);
  INFO_SET(mapsize);
  INFO_SET(last_pgno);
  INFO_SET(last_txnid);
  INFO_SET(maxreaders);
  INFO_SET(numreaders);
#undef INFO_SET

  return ret;
}

#mapsize=(size) ⇒ Object 



1056
1057
1058
1059
1060
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1056

static VALUE environment_set_mapsize(VALUE self, VALUE size) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  check(mdb_env_set_mapsize(environment->env, NUM2LONG(size)));
  return Qnil;
}

#path ⇒ String

Return the path to the database environment files

Returns:

  • (String) —

    the path that was used to open the environment.



1049
1050
1051
1052
1053
1054
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1049

static VALUE environment_path(VALUE self) {
  const char* path;
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  check(mdb_env_get_path(environment->env, &path));
  return rb_str_new2(path);
}

#reader_check ⇒ Object 



782
783
784
785
786
787
788
789
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 782

static VALUE environment_reader_check(VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);

  int dead = 0;
  mdb_reader_check(environment->env, &dead);

  return INT2NUM(dead);
}

#reader_list ⇒ Object 



771
772
773
774
775
776
777
778
779
780
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 771

static VALUE environment_reader_list(VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);

  VALUE ret = rb_ary_new();

  mdb_reader_list(environment->env, reader_list_callback, (void*)ret);

  //return Qnil;
  return ret;
}

#set_flags(flags) ⇒ Object

Set one or more flags in the environment. The available flags are defined in #flags.

Examples:

env.set_flags :nosync, :writemap

Parameters:

  • flags (Array) —

    Array of flag names (symbols) to set

Returns:

  • nil

Raises:

  • (Error) —

    if an invalid flag name is specified

See Also:



1089
1090
1091
1092
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1089

static VALUE environment_set_flags(int argc, VALUE* argv, VALUE self) {
  environment_change_flags(argc, argv, self, 1);
  return Qnil;
}

#stat ⇒ Hash

Return useful statistics about an environment.

  • :psize Size of a database page

  • :depth Depth (height) of the B-tree

  • :branch_pages Number of internal (non-leaf) pages

  • :leaf_pages Number of leaf pages

  • :overflow_pages Number of overflow pages

  • :entries Number of data items

Returns:

  • (Hash) —

    the statistics



827
828
829
830
831
832
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 827

static VALUE environment_stat(VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);
  MDB_stat stat;
  check(mdb_env_stat(environment->env, &stat));
  return stat2hash(&stat);
}

#sync(force) ⇒ Object

Flush the data buffers to disk.

Data is always written to disk when Transaction#commit is called, but the operating system may keep it buffered. MDB always flushes the OS buffers upon commit as well, unless the environment was opened with :nosync or in part :nometasync.

Parameters:

  • force (Boolean) —

    If true, force a synchronous flush. Otherwise if the environment has the :nosync flag set the flushes will be omitted, and with :mapasync they will be asynchronous.



897
898
899
900
901
902
903
904
905
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 897

static VALUE environment_sync(int argc, VALUE *argv, VALUE self) {
  ENVIRONMENT(self, &lmdb_environment_type, environment);

  VALUE force;
  rb_scan_args(argc, argv, "01", &force);

  check(mdb_env_sync(environment->env, RTEST(force)));
  return Qnil;
}

#transaction(readonly) {|txn| ... } ⇒ Object

Note:

Transactions can be nested.

Note:

do a ‘break txn.commit` if you want to exit from the block early without aborting.

Begin a transaction. Takes a block to run the body of the transaction. A transaction commits when it exits the block successfully. A transaction aborts when it raises an exception or calls Transaction#abort.

Examples:

db = env.database "mydata"
env.transaction do |txn1|
  db['a'] = 1
  env.transaction do |txn2|
    # txn2 is nested in txn1
    db['a'] = 2
    db['a']                    #=> 2
    txn2.abort
  end
  db['a']                      #=> 1
  env.transaction do
    db['a'] = 3
  end
end
db['a']                        #=> 3

Parameters:

  • readonly (Boolean) —

    This transaction will not perform any write operations

Yields:

  • (txn) —

    The block to be executed with the body of the transaction.

Yield Parameters:

  • txn (Transaction) —

    An optional transaction argument



1221
1222
1223
1224
1225
1226
1227
1228
1229
 
# File 'ext/lmdb_ext/lmdb_ext.c', line 1221

static VALUE environment_transaction(int argc, VALUE *argv, VALUE self) {
  rb_need_block();

  VALUE readonly;
  rb_scan_args(argc, argv, "01", &readonly);
  unsigned int flags = RTEST(readonly) ? MDB_RDONLY : 0;

  return with_transaction(self, rb_yield, Qnil, flags);
}