Class: CAFrame::CSVParser::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/carray/frame/csv_parser.rb

Overview

Splits records into fields. Regexps are compiled once and reused across every record, so per-row cost stays low.

Instance Method Summary collapse

Constructor Details

#initialize(sep, quote, strip) ⇒ Tokenizer

Returns a new instance of Tokenizer.



83
84
85
86
87
88
89
90
91
92
 
# File 'lib/carray/frame/csv_parser.rb', line 83

def initialize(sep, quote, strip)
  @sep      = sep
  @quote    = quote
  @strip    = strip
  @sep_re   = /#{Regexp.escape(sep)}/
  @quote_re = /#{Regexp.escape(quote)}/
  @unquoted = /[^#{Regexp.escape(sep)}]*/
  @inner    = /[^#{Regexp.escape(quote)}]*/
  @recno    = 0
end

Instance Method Details

#read(io) ⇒ Object

Fields of the next non-blank record, or nil at EOF.



95
96
97
98
99
100
101
102
103
 
# File 'lib/carray/frame/csv_parser.rb', line 95

def read(io)
  loop do
    rec = CSVParser.read_record(io, @quote)
    return nil if rec.nil?
    @recno += 1
    next if rec.empty?
    return rec.count(@quote).zero? ? simple(rec) : scan(rec)
  end
end