Class: Classifier::Streaming::LineReader

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/classifier/streaming/line_reader.rb

Overview

Memory-efficient line reader for large files and IO streams. Reads lines one at a time and can yield in configurable batches.

Examples:

Reading line by line

reader = LineReader.new(File.open('large_corpus.txt'))
reader.each { |line| process(line) }

Reading in batches

reader = LineReader.new(io, batch_size: 100)
reader.each_batch { |batch| process_batch(batch) }

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(io, batch_size: 100) ⇒ LineReader

Creates a new LineReader.

RBS:

  • (IO | Classifier::Streaming::MultiIO, ?batch_size: Integer) -> void



26
27
28
29
 
# File 'lib/classifier/streaming/line_reader.rb', line 26

def initialize(io, batch_size: 100)
  @io = io
  @batch_size = batch_size
end

Instance Attribute Details

#batch_size ⇒ Object (readonly)

RBS:

  • @io: IO | Classifier::Streaming::MultiIO

  • @batch_size: Integer



21
22
23
 
# File 'lib/classifier/streaming/line_reader.rb', line 21

def batch_size
  @batch_size
end

Instance Method Details

#each ⇒ Object

Iterates over each line in the IO stream. Lines are chomped (trailing newlines removed).

RBS:

  • () { (String) -> void } -> void

  • () -> Enumerator[String, void]



36
37
38
39
40
41
42
 
# File 'lib/classifier/streaming/line_reader.rb', line 36

def each
  return enum_for(:each) unless block_given?

  @io.each_line do |line|
    yield line.chomp
  end
end

#each_batch {|batch| ... } ⇒ Object

Iterates over batches of lines. Each batch is an array of chomped lines.

RBS:

  • () { (Array[String]) -> void } -> void

  • () -> Enumerator[Array[String], void]

Yields:

  • (batch) 


49
50
51
52
53
54
55
56
57
58
59
60
61
 
# File 'lib/classifier/streaming/line_reader.rb', line 49

def each_batch
  return enum_for(:each_batch) unless block_given?

  batch = [] #: Array[String]
  each do |line|
    batch << line
    if batch.size >= @batch_size
      yield batch
      batch = []
    end
  end
  yield batch unless batch.empty?
end

#estimate_line_count(sample_size: 100) ⇒ Object

Estimates the total number of lines in the IO stream. This is a rough estimate based on file size and average line length. Returns nil for non-seekable streams.

RBS:

  • (?sample_size: Integer) -> Integer?



68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/classifier/streaming/line_reader.rb', line 68

def estimate_line_count(sample_size: 100)
  return nil unless @io.respond_to?(:size) && @io.respond_to?(:rewind)

  io = @io #: ::IO

  begin
    original_pos = io.pos
    io.rewind

    sample_bytes = 0
    sample_lines = 0

    sample_size.times do
      line = io.gets
      break unless line

      sample_bytes += line.bytesize
      sample_lines += 1
    end

    io.seek(original_pos)

    return nil if sample_lines.zero?

    avg_line_size = sample_bytes.to_f / sample_lines
    io_size = io.__send__(:size) #: Integer
    (io_size / avg_line_size).round
  rescue IOError, Errno::ESPIPE
    nil
  end
end