Module: ActiveModel::Datastore::ClassMethods

Defined in:
lib/active_model/datastore.rb

Overview

Methods defined here will be class methods when 'include ActiveModel::Datastore'.

Instance Method Summary collapse

Instance Method Details

#all(options = {}) ⇒ Array<Model>, String

Queries entities from Cloud Datastore by named kind and using the provided options. When a limit option is provided queries up to the limit and returns results with a cursor.

This method may make several API calls until all query results are retrieved. The run method returns a QueryResults object, which is a special case Array with additional values. QueryResults are returned in batches, and the batch size is determined by the Datastore API. Batch size is not guaranteed. It will be affected by the size of the data being returned, and by other forces such as how distributed and/or consistent the data in Datastore is. Calling all on the QueryResults retrieves all results by repeatedly loading #next until #next? returns false. The all method returns an enumerator which from_entities iterates on.

Be sure to use as narrow a search criteria as possible. Please use with caution.

or if options was provided:

Parameters:

  • options (Hash) (defaults to: {}) —

    The options to construct the query with.

Options Hash (options):

  • :ancestor (Google::Cloud::Datastore::Key) —

    Filter for inherited results.

  • :cursor (String) —

    Sets the cursor to start the results at.

  • :limit (Integer) —

    Sets a limit to the number of results to be returned.

  • :order (String) —

    Sort the results by property name.

  • :desc_order (String) —

    Sort the results by descending property name.

  • :select (Array) —

    Retrieve only select properties from the matched entities.

  • :distinct_on (Array) —

    Group results by a list of properties.

  • :where (Array) —

    Adds a property filter of arrays in the format [name, operator, value].

Returns:

  • (Array<Model>, String) —

    An array of ActiveModel results

  • (Array<Model>, String) —

    An array of ActiveModel results and a cursor that can be used to query for additional results.



300
301
302
303
304
305
306
307
308
309
310
311
 
# File 'lib/active_model/datastore.rb', line 300

def all(options = {})
  next_cursor = nil
  query = build_query(options)
  query_results = retry_on_exception(operation: 'run query') do
    CloudDatastore.dataset.run query
  end
  if options[:limit]
    next_cursor = query_results.cursor if query_results.size == options[:limit]
    return from_entities(query_results.all), next_cursor
  end
  from_entities(query_results.all)
end

#build_model(entity) ⇒ Object 



527
528
529
530
531
532
533
 
# File 'lib/active_model/datastore.rb', line 527

def build_model(entity)
  model_entity = new
  model_entity.id = entity.key.id unless entity.key.id.nil?
  model_entity.id = entity.key.name unless entity.key.name.nil?
  model_entity.parent_key_id = entity.key.parent.id if entity.key.parent.present?
  model_entity
end

#build_query(options = {}) ⇒ Query

Constructs a Google::Cloud::Datastore::Query.

Parameters:

  • options (Hash) (defaults to: {}) —

    The options to construct the query with.

Options Hash (options):

  • :ancestor (Google::Cloud::Datastore::Key) —

    Filter for inherited results.

  • :cursor (String) —

    Sets the cursor to start the results at.

  • :limit (Integer) —

    Sets a limit to the number of results to be returned.

  • :order (String) —

    Sort the results by property name.

  • :desc_order (String) —

    Sort the results by descending property name.

  • :select (Array) —

    Retrieve only select properties from the matched entities.

  • :distinct_on (Array) —

    Group results by a list of properties.

  • :where (Array) —

    Adds a property filter of arrays in the format [name, operator, value].

Returns:

  • (Query) —

    A datastore query.



412
413
414
415
 
# File 'lib/active_model/datastore.rb', line 412

def build_query(options = {})
  query = CloudDatastore.dataset.query name
  query_options(query, options)
end

#find(*ids, parent: nil) ⇒ Model, ...

Find entity by id - this can either be a specific id (1), a list of ids (1, 5, 6), or an array of ids ([5, 6, 10]). The parent key is optional.

Parameters:

  • ids (Integer) —

    One or more ids to retrieve.

  • parent (Google::Cloud::Datastore::Key) (defaults to: nil) —

    The parent key of the entity.

Returns:

  • (Model, nil) —

    An ActiveModel object or nil for a single id.

  • (Array<Model>) —

    An array of ActiveModel objects for more than one id.



323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
 
# File 'lib/active_model/datastore.rb', line 323

def find(*ids, parent: nil)
  expects_array = ids.first.is_a?(Array)
  ids = ids.flatten.compact.uniq.map(&:to_i)

  case ids.size
  when 0
    raise EntityError, "Couldn't find #{name} without an ID"
  when 1
    entity = find_entity(ids.first, parent)
    model_entity = from_entity(entity)
    expects_array ? [model_entity].compact : model_entity
  else
    lookup_results = find_all_entities(ids, parent)
    from_entities(lookup_results.all)
  end
end

#find_all_entities(ids_or_names, parent) ⇒ Object

Finds entities by keys using the provided array items. Results provided by the dataset find_all is a Dataset::LookupResults object.

Parameters:

  • ids_or_names (Array<Integer>, Array<String>) —

    An array of ids or names.



521
522
523
524
525
 
# File 'lib/active_model/datastore.rb', line 521

def find_all_entities(ids_or_names, parent)
  keys = ids_or_names.map { |id| CloudDatastore.dataset.key name, id }
  keys.map { |key| key.parent = parent } if parent.present?
  retry_on_exception(operation: 'find all') { CloudDatastore.dataset.find_all keys }
end

#find_by(args) ⇒ Model?

Finds the first entity matching the specified condition.

Examples:

User.find_by(name: 'Joe')
User.find_by(name: 'Bryce', ancestor: parent_key)

Parameters:

  • args (Hash) —

    In which the key is the property and the value is the value to look for.

Options Hash (args):

  • :ancestor (Google::Cloud::Datastore::Key) —

    filter for inherited results

Returns:

  • (Model, nil) —

    An ActiveModel object or nil.



352
353
354
355
356
357
358
359
360
361
 
# File 'lib/active_model/datastore.rb', line 352

def find_by(args)
  query = CloudDatastore.dataset.query name
  query.ancestor(args[:ancestor]) if args[:ancestor]
  query.limit(1)
  query.where(args.keys[0].to_s, '=', args.values[0])
  query_results = retry_on_exception(operation: 'run query') do
    CloudDatastore.dataset.run query
  end
  from_entity(query_results.first)
end

#find_entities(*ids_or_names, parent: nil) ⇒ Array<Entity>

Retrieves the entities for the provided ids by key and by an optional parent. The find_all method returns LookupResults, which is a special case Array with additional values. LookupResults are returned in batches, and the batch size is determined by the Datastore API. Batch size is not guaranteed. It will be affected by the size of the data being returned, and by other forces such as how distributed and/or consistent the data in Datastore is. Calling all on the LookupResults retrieves all results by repeatedly loading #next until #next? returns false. The all method returns an enumerator unless passed a block. We iterate on the enumerator to return the model entity objects.

Parameters:

  • ids_or_names (Integer, String) —

    One or more ids to retrieve.

  • parent (Google::Cloud::Datastore::Key) (defaults to: nil) —

    The parent Key of the entity.

Returns:

  • (Array<Entity>) —

    an array of Google::Cloud::Datastore::Entity objects.



261
262
263
264
265
 
# File 'lib/active_model/datastore.rb', line 261

def find_entities(*ids_or_names, parent: nil)
  ids_or_names = ids_or_names.flatten.compact.uniq
  lookup_results = find_all_entities(ids_or_names, parent)
  lookup_results.all.collect { |x| x }
end

#find_entity(id_or_name, parent = nil) ⇒ Entity?

Retrieves an entity by id or name and by an optional parent.

Parameters:

  • id_or_name (Integer or String) —

    The id or name value of the entity Key.

  • parent (Google::Cloud::Datastore::Key) (defaults to: nil) —

    The parent Key of the entity.

Returns:

  • (Entity, nil) —

    a Google::Cloud::Datastore::Entity object or nil.



239
240
241
242
243
 
# File 'lib/active_model/datastore.rb', line 239

def find_entity(id_or_name, parent = nil)
  key = CloudDatastore.dataset.key name, id_or_name
  key.parent = parent if parent.present?
  retry_on_exception(operation: 'find') { CloudDatastore.dataset.find key }
end

#from_entities(entities) ⇒ Object

Translates an Enumerator of Datastore::Entity objects to ActiveModel::Model objects.

Results provided by the dataset find_all or run query will be a Dataset::LookupResults or Dataset::QueryResults object. Invoking all on those objects returns an enumerator.

Parameters:

  • entities (Enumerator) —

    An enumerator representing the datastore entities.

Raises:

  • (ArgumentError) 


371
372
373
374
375
 
# File 'lib/active_model/datastore.rb', line 371

def from_entities(entities)
  raise ArgumentError, 'Entities param must be an Enumerator' unless entities.is_a? Enumerator

  entities.map { |entity| from_entity(entity) }
end

#from_entity(entity) ⇒ Model

Translates between Datastore::Entity objects and ActiveModel::Model objects.

Parameters:

  • entity (Entity) —

    Entity from Cloud Datastore.

Returns:

  • (Model) —

    The translated ActiveModel object.



383
384
385
386
387
388
389
390
391
392
393
 
# File 'lib/active_model/datastore.rb', line 383

def from_entity(entity)
  return if entity.nil?

  model_entity = build_model(entity)
  model_entity.entity_property_values = entity.properties.to_h
  entity.properties.to_h.each do |name, value|
    model_entity.send "#{name}=", value if model_entity.respond_to? "#{name}="
  end
  model_entity.reload!
  model_entity
end

#log_google_cloud_error ⇒ Object 



467
468
469
470
471
472
 
# File 'lib/active_model/datastore.rb', line 467

def log_google_cloud_error
  yield
rescue Google::Cloud::Error => e
  puts "\e[33m[#{e.message.inspect}]\e[0m"
  raise e
end

#parent_key(parent_id) ⇒ Object

A default parent key for specifying an ancestor path and creating an entity group.



227
228
229
 
# File 'lib/active_model/datastore.rb', line 227

def parent_key(parent_id)
  CloudDatastore.dataset.key('Parent' + name, parent_id.to_i)
end

#query_options(query, options) ⇒ Object

**************** private ****************



476
477
478
479
480
481
482
483
484
 
# File 'lib/active_model/datastore.rb', line 476

def query_options(query, options)
  query.ancestor(options[:ancestor]) if options[:ancestor]
  query.cursor(options[:cursor]) if options[:cursor]
  query.limit(options[:limit]) if options[:limit]
  query_sort(query, options)
  query.select(*options[:select]) if options[:select]
  query.distinct_on(*options[:distinct_on]) if options[:distinct_on]
  query_property_filter(query, options)
end

#query_property_filter(query, options) ⇒ Object

Adds property filters to the query if included in the options. Accepts individual or nested Arrays:

[['superseded', '=', false], ['email', '=', 'something']]


500
501
502
503
504
505
506
507
508
509
510
511
512
 
# File 'lib/active_model/datastore.rb', line 500

def query_property_filter(query, options)
  if options[:where]
    opts = options[:where]
    if opts[0].is_a?(Array)
      opts.each do |opt|
        query.where(opt[0], opt[1], opt[2]) unless opt.nil?
      end
    else
      query.where(opts[0], opts[1], opts[2])
    end
  end
  query
end

#query_sort(query, options) ⇒ Object

Adds sorting to the results by a property name if included in the options.



489
490
491
492
493
 
# File 'lib/active_model/datastore.rb', line 489

def query_sort(query, options)
  query.order(options[:order]) if options[:order]
  query.order(options[:desc_order], :desc) if options[:desc_order]
  query
end

#retry_on_exception(max_retry_count = 5, operation: nil, kind: name) ⇒ Object 



442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
 
# File 'lib/active_model/datastore.rb', line 442

def retry_on_exception(max_retry_count = 5, operation: nil, kind: name)
  retries = 0
  sleep_time = 0.25
  begin
    started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
    yield
  rescue Google::Cloud::Error => e
    raise e if retries >= max_retry_count

    if operation
      elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
      message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
                "#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
    else
      message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m"
    end
    ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
    # 0.25, 0.5, 1, 2, and 4 second between retries.
    sleep sleep_time
    retries += 1
    sleep_time *= 2
    retry
  end
end

#retry_on_exception?(max_retry_count = 5, operation: nil, kind: name) ⇒ Boolean

Returns:

  • (Boolean) 


417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
 
# File 'lib/active_model/datastore.rb', line 417

def retry_on_exception?(max_retry_count = 5, operation: nil, kind: name)
  retries = 0
  sleep_time = 0.25
  begin
    started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC) if operation
    yield
  rescue Google::Cloud::Error => e
    return false if retries >= max_retry_count

    if operation
      elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at) * 1000).round
      message = "\e[33mDatastore #{operation} failed for #{kind} after #{elapsed_ms} ms: " \
                "#{e.message.inspect}; retrying in #{sleep_time} s\e[0m"
    else
      message = "\e[33mRescued exception #{e.message.inspect}, retrying in #{sleep_time}\e[0m"
    end
    ActiveModel::Datastore.logger ? ActiveModel::Datastore.logger.warn(message) : puts(message)
    # 0.25, 0.5, 1, 2, and 4 second between retries.
    sleep sleep_time
    retries += 1
    sleep_time *= 2
    retry
  end
end