Class: JobIteration::EnumeratorBuilder

Inherits:
Object
  • Object
show all
Extended by:
Forwardable
Defined in:
lib/job-iteration/enumerator_builder.rb

Defined Under Namespace

Classes: Wrapper

Instance Method Summary collapse

Constructor Details

#initialize(job, wrapper: Wrapper) ⇒ EnumeratorBuilder

Returns a new instance of EnumeratorBuilder.



31
32
33
34
 
# File 'lib/job-iteration/enumerator_builder.rb', line 31

def initialize(job, wrapper: Wrapper)
  @job = job
  @wrapper = wrapper
end

Instance Method Details

#build_active_record_enumerator_on_batch_relations(scope, wrap: true, cursor:, **args) ⇒ Object Also known as: active_record_on_batch_relations

Builds Enumerator from Active Record Relation and enumerates on batches, yielding Active Record Relations. See documentation for #build_active_record_enumerator_on_batches.



122
123
124
125
126
127
128
129
130
 
# File 'lib/job-iteration/enumerator_builder.rb', line 122

def build_active_record_enumerator_on_batch_relations(scope, wrap: true, cursor:, **args)
  enum = JobIteration::ActiveRecordBatchEnumerator.new(
    scope,
    cursor: cursor,
    **args
  ).each
  enum = wrap(self, enum) if wrap
  enum
end

#build_active_record_enumerator_on_batches(scope, cursor:, **args) ⇒ Object Also known as: active_record_on_batches

Builds Enumerator from Active Record Relation and enumerates on batches of records. Each Enumerator tick moves the cursor +batch_size+ rows forward.

+batch_size:+ sets how many records will be fetched in one batch. Defaults to 100.

For the rest of arguments, see documentation for #build_active_record_enumerator_on_records



111
112
113
114
115
116
117
118
 
# File 'lib/job-iteration/enumerator_builder.rb', line 111

def build_active_record_enumerator_on_batches(scope, cursor:, **args)
  enum = build_active_record_enumerator(
    scope,
    cursor: cursor,
    **args
  ).batches
  wrap(self, enum)
end

#build_active_record_enumerator_on_records(scope, cursor:, **args) ⇒ Object Also known as: active_record_on_records

Builds Enumerator from Active Record Relation. Each Enumerator tick moves the cursor one row forward.

+columns:+ argument is used to build the actual query for iteration. +columns+: defaults to primary key:

1) SELECT * FROM users ORDER BY id LIMIT 100

When iteration is resumed, +cursor:+ and +columns:+ values will be used to continue from the point where iteration stopped:

2) SELECT * FROM users WHERE id > $CURSOR ORDER BY id LIMIT 100

+columns:+ can also take more than one column. In that case, +cursor+ will contain serialized values of all columns at the point where iteration stopped.

Consider this example with +columns: [:created_at, :id]+. Here's the query will use on the first iteration:

1) SELECT * FROM products ORDER BY created_at, id LIMIT 100

And the query on the next iteration:

2) SELECT * FROM products WHERE (created_at > '$LAST_CREATED_AT_CURSOR' OR (created_at = '$LAST_CREATED_AT_CURSOR' AND (id > '$LAST_ID_CURSOR'))) ORDER BY created_at, id LIMIT 100

As a result of this query pattern, if the values in these columns change for the records in scope during iteration, they may be skipped or yielded multiple times depending on the nature of the update and the cursor's value. If the value gets updated to a greater value than the cursor's value, it will get yielded again. Similarly, if the value gets updated to a lesser value than the curor's value, it will get skipped.



96
97
98
99
100
101
102
103
 
# File 'lib/job-iteration/enumerator_builder.rb', line 96

def build_active_record_enumerator_on_records(scope, cursor:, **args)
  enum = build_active_record_enumerator(
    scope,
    cursor: cursor,
    **args
  ).records
  wrap(self, enum)
end

#build_array_enumerator(enumerable, cursor:) ⇒ Object Also known as: array

Builds Enumerator object from a given array, using +cursor+ as an offset.



50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
 
# File 'lib/job-iteration/enumerator_builder.rb', line 50

def build_array_enumerator(enumerable, cursor:)
  unless enumerable.is_a?(Array)
    raise ArgumentError, "enumerable must be an Array"
  end
  if enumerable.any? { |i| defined?(ActiveRecord) && i.is_a?(ActiveRecord::Base) }
    raise ArgumentError, "array cannot contain ActiveRecord objects"
  end
  drop =
    if cursor.nil?
      0
    else
      cursor + 1
    end

  wrap(self, enumerable.each_with_index.drop(drop).to_enum { enumerable.size })
end

#build_once_enumerator(cursor:) ⇒ Object Also known as: once

Builds Enumerator objects that iterates once.



39
40
41
 
# File 'lib/job-iteration/enumerator_builder.rb', line 39

def build_once_enumerator(cursor:)
  wrap(self, build_times_enumerator(1, cursor: cursor))
end

#build_throttle_enumerator(enum, throttle_on:, backoff:) ⇒ Object Also known as: throttle 



132
133
134
135
136
137
138
139
 
# File 'lib/job-iteration/enumerator_builder.rb', line 132

def build_throttle_enumerator(enum, throttle_on:, backoff:)
  JobIteration::ThrottleEnumerator.new(
    enum,
    @job,
    throttle_on: throttle_on,
    backoff: backoff
  ).to_enum
end

#build_times_enumerator(number, cursor:) ⇒ Object Also known as: times

Builds Enumerator objects that iterates N times and yields number starting from zero.

Raises:

  • (ArgumentError) 


44
45
46
47
 
# File 'lib/job-iteration/enumerator_builder.rb', line 44

def build_times_enumerator(number, cursor:)
  raise ArgumentError, "First argument must be an Integer" unless number.is_a?(Integer)
  wrap(self, build_array_enumerator(number.times.to_a, cursor: cursor))
end