Class: DBF::RecordIterator

Inherits:
Object
  • Object
show all
Defined in:
lib/dbf/record_iterator.rb

Constant Summary collapse

CHUNK_SIZE =

Records are read in chunks of whole records totalling roughly this many bytes, so enumerating a multi-gigabyte file (for example a shapefile sidecar) needs only chunk-sized memory instead of the entire record section at once.

4 * 1024 * 1024

Instance Method Summary collapse

Constructor Details

#initialize(data, context, header_length, record_length, record_count, chunk_size: CHUNK_SIZE) ⇒ RecordIterator

Returns a new instance of RecordIterator.



11
12
13
14
15
16
17
18
 
# File 'lib/dbf/record_iterator.rb', line 11

def initialize(data, context, header_length, record_length, record_count, chunk_size: CHUNK_SIZE)
  @data = data
  @context = context
  @header_length = header_length
  @record_length = record_length
  @record_count = record_count
  @chunk_size = chunk_size
end

Instance Method Details

#each ⇒ Object 



20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/dbf/record_iterator.rb', line 20

def each(&)
  return enum_for(:each) unless block_given?

  # A record_length of 0 from a crafted header cannot drive an unbounded
  # loop: capacity is 0 and enumeration ends immediately.
  remaining = record_capacity
  @data.seek(@header_length)

  while remaining.positive?
    wanted = [per_chunk, remaining].min
    buffer = @data.read(wanted * @record_length)
    break unless buffer

    whole_records = buffer.bytesize / @record_length
    break if whole_records.zero?

    yield_chunk(buffer, whole_records, &)
    remaining -= whole_records

    # A short read means the file ended earlier than the header promised
    break if whole_records < wanted
  end
end