Class: Leptris::XML::SAX::Recorder

Inherits:
Object
  • Object
show all
Defined in:
lib/leptris/xml/sax/recorder.rb

Overview

Chunked event recorder (libleptris 1.9.4, issue #585): a SAX transport that buffers events C-side — fixed LeptrisSaxEventRecord entries plus a packed string arena — and drains them in one bulk read per fed chunk. The handler receives the identical calls the callback API makes; only the transport changes, and the callback count becomes O(chunks), not O(events): through FFI, per-event C-to-Ruby dispatch cost more than the parse itself.

Leptris::XML::SAX::Recorder.parse(xml) do |kind, name, text, attrs|
  # kind: :start_element, :characters, ... (see KINDS)
end

Constant Summary collapse

KINDS = 
{
  Leptris::XML::FFI::SAX_EVENT_START_DOCUMENT => :start_document,
  Leptris::XML::FFI::SAX_EVENT_END_DOCUMENT => :end_document,
  Leptris::XML::FFI::SAX_EVENT_START_ELEMENT => :start_element,
  Leptris::XML::FFI::SAX_EVENT_END_ELEMENT => :end_element,
  Leptris::XML::FFI::SAX_EVENT_CHARACTERS => :characters,
  Leptris::XML::FFI::SAX_EVENT_COMMENT => :comment,
  Leptris::XML::FFI::SAX_EVENT_CDATA => :cdata,
  Leptris::XML::FFI::SAX_EVENT_PI => :pi,
  Leptris::XML::FFI::SAX_EVENT_START_PREFIX => :start_prefix,
  Leptris::XML::FFI::SAX_EVENT_END_PREFIX => :end_prefix,
  Leptris::XML::FFI::SAX_EVENT_ERROR => :error,
}.freeze
KIND_BY_CODE =

Kind symbol per code byte — Array indexing in the drain loop, not a Hash lookup per event.

Array.new(KINDS.size) { |code| KINDS[code] }.freeze
HAS_NAME =

Which kinds carry each string field (the record layout leaves the other at offset 0 / length 0): slicing a zero-length run allocates a wasted empty String per event — 150k+ of them on a text-heavy document. Unused fields yield nil (the callback transport delivers no name for characters either).

{
  start_element: true, end_element: true,
  pi: true, start_prefix: true, end_prefix: true,
}.freeze
HAS_TEXT = 
{
  characters: true, comment: true, cdata: true,
  pi: true, start_prefix: true, error: true,
}.freeze
RECORD =

Layout offsets (the struct stays the ABI's single source of truth) and per-count kind-strip templates: one byte per record, skipping the stride — the kind codes are 0..10, so the strip unpacks to cache-resident Fixnums, unlike the old whole-record unpack whose 2.25M-Integer array for a single-feed drain was most of the loop's allocation profile.

Leptris::XML::FFI::SaxEventRecord
NAME_OFF = 
RECORD.offset_of(:name_off)
NAME_LEN = 
RECORD.offset_of(:name_len)
TEXT_OFF = 
RECORD.offset_of(:text_off)
TEXT_LEN = 
RECORD.offset_of(:text_len)
ATTRS_OFF = 
RECORD.offset_of(:attrs_off)
ATTR_COUNT = 
RECORD.offset_of(:attr_count)
LINE_OFF = 
RECORD.offset_of(:line)
COLUMN_OFF = 
RECORD.offset_of(:column)
RECORD_STRIDE = 
RECORD.size
SAX_EVENT_CHARACTERS_CODE = 
Leptris::XML::FFI::SAX_EVENT_CHARACTERS
NUL = 
"\0".freeze
KIND_TEMPLATE_FOR = 
::Hash.new do |cache, count|
  cache[count] = ("C x#{RECORD_STRIDE - 1}" * count)
end

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(handle) ⇒ Recorder

Returns a new instance of Recorder.



62
63
64
 
# File 'lib/leptris/xml/sax/recorder.rb', line 62

def initialize(handle)
  @handle = handle
end

Class Method Details

.open ⇒ Object 



36
37
38
39
40
41
42
 
# File 'lib/leptris/xml/sax/recorder.rb', line 36

def self.open
  raw = Leptris::XML::FFI.leptris_sax_recorder_new
  if raw.null?
    raise Leptris::XML::Error, "leptris_sax_recorder_new failed"
  end
  new(raw)
end

.parse(xml_or_io, kinds: nil) ⇒ Object

One-shot parse over a complete document, yielding Record structs. kinds filters the drain (see #each_event) — pass e.g. kinds: [:start_element] to slice strings only for the events you consume.



48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/leptris/xml/sax/recorder.rb', line 48

def self.parse(xml_or_io, kinds: nil)
  recorder = open
  begin
    if xml_or_io.respond_to?(:read)
      recorder.feed_stream(xml_or_io, kinds: kinds) { |*args| yield(*args) }
    else
      recorder.feed(xml_or_io.to_s, final: true)
      recorder.each_event(*Array(kinds)) { |*args| yield(*args) }
    end
  ensure
    recorder.free
  end
end

Instance Method Details

#attrs_pairs(arena, off, count) ⇒ Object

attrs as [name, value] pairs in source order — the callback transport's shape (walk_attr_array).



239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
 
# File 'lib/leptris/xml/sax/recorder.rb', line 239

def attrs_pairs(arena, off, count)
  return [] if count.zero?
  pairs = []
  pos = off
  nul = NUL
  utf8 = Encoding::UTF_8
  while count > 0
    name_end = arena.index(nul, pos)
    name = arena.byteslice(pos, name_end - pos).force_encoding(utf8)
    pos = name_end + 1
    value_end = arena.index(nul, pos)
    pairs << [name,
      arena.byteslice(pos, value_end - pos).force_encoding(utf8)]
    pos = value_end + 1
    count -= 1
  end
  pairs
end

#dispatch(handler, dispatched) ⇒ Object

Dispatch drain: delivers the recorded events as handler method calls with the CALLBACK transport's exact shapes — pairs arrays (not hashes), one-argument start_element arity dispatch, UTF-8 strings, PI-data whitespace normalization. dispatched maps kind symbols to true for the kinds the handler overrides (the attach-only-overridden policy carried onto the bulk transport: other records cost one strip read). The C recorder costs ~12ms per 250k-event document while 250k ffi callback trampolines cost ~150ms — this is how a many-override handler beats the callback transport (and Nokogiri's C extension) outright.



149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
 
# File 'lib/leptris/xml/sax/recorder.rb', line 149

def dispatch(handler, dispatched)
  count_ptr = ::FFI::MemoryPointer.new(:size_t)
  len_ptr = ::FFI::MemoryPointer.new(:size_t)
  begin
    records_ptr = Leptris::XML::FFI.leptris_sax_recorder_records(
      @handle, count_ptr)
    return self if records_ptr.null? || count_ptr.read_uint64.zero?
    arena_ptr = Leptris::XML::FFI.leptris_sax_recorder_arena(
      @handle, len_ptr)
    arena = arena_ptr.read_bytes(len_ptr.read_uint64)
    count = count_ptr.read_uint64
    records = records_ptr.read_bytes(count * RECORD_STRIDE)
    kinds = records.unpack(KIND_TEMPLATE_FOR[count])

    one_arg_start = dispatched[:start_element] == :one_arg
    i = 0
    while i < count
      code = kinds[i]
      kind = KIND_BY_CODE[code]
      if kind && dispatched[kind]
        base = i * RECORD_STRIDE
        case kind
        when :characters
          handler.characters(slice(arena,
            records_ptr.get_uint32(base + TEXT_OFF),
            records_ptr.get_uint32(base + TEXT_LEN)))
        when :start_element
          name = slice(arena,
            records_ptr.get_uint32(base + NAME_OFF),
            records_ptr.get_uint32(base + NAME_LEN))
          if one_arg_start
            handler.start_element(name)
          else
            handler.start_element(name, attrs_pairs(arena,
              records_ptr.get_uint32(base + ATTRS_OFF),
              records_ptr.get_uint32(base + ATTR_COUNT)))
          end
        when :end_element
          handler.end_element(slice(arena,
            records_ptr.get_uint32(base + NAME_OFF),
            records_ptr.get_uint32(base + NAME_LEN)))
        when :comment
          handler.comment(slice(arena,
            records_ptr.get_uint32(base + TEXT_OFF),
            records_ptr.get_uint32(base + TEXT_LEN)))
        when :cdata
          handler.cdata_block(slice(arena,
            records_ptr.get_uint32(base + TEXT_OFF),
            records_ptr.get_uint32(base + TEXT_LEN)))
        when :pi
          handler.processing_instruction(slice(arena,
            records_ptr.get_uint32(base + NAME_OFF),
            records_ptr.get_uint32(base + NAME_LEN)),
            Leptris::XML::FFI.read_pi_data(slice(arena,
              records_ptr.get_uint32(base + TEXT_OFF),
              records_ptr.get_uint32(base + TEXT_LEN))))
        when :start_prefix
          handler.start_prefix_mapping(slice(arena,
            records_ptr.get_uint32(base + NAME_OFF),
            records_ptr.get_uint32(base + NAME_LEN)),
            slice(arena,
              records_ptr.get_uint32(base + TEXT_OFF),
              records_ptr.get_uint32(base + TEXT_LEN)))
        when :end_prefix
          handler.end_prefix_mapping(slice(arena,
            records_ptr.get_uint32(base + NAME_OFF),
            records_ptr.get_uint32(base + NAME_LEN)))
        when :error
          handler.error(slice(arena,
            records_ptr.get_uint32(base + TEXT_OFF),
            records_ptr.get_uint32(base + TEXT_LEN)),
            records_ptr.get_uint32(base + LINE_OFF),
            records_ptr.get_uint32(base + COLUMN_OFF))
        when :start_document
          handler.start_document
        when :end_document
          handler.end_document
        end
      end
      i += 1
    end
  ensure
    count_ptr.free
    len_ptr.free
  end
  self
end

#each_event(*kinds) ⇒ Object

Drains the current chunk's records: bulk-reads the record array and the packed arena (two FFI calls), then slices every string from the arena in Ruby — no per-string FFI. With kinds, records of other kinds are skipped BEFORE any string is sliced: an unwanted event costs one Array read, and cost scales with what the consumer asked for, not with the document's event mix.



264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
 
# File 'lib/leptris/xml/sax/recorder.rb', line 264

def each_event(*kinds)
  wanted = kinds.empty? ? nil : wanted_set(kinds)
  count_ptr = ::FFI::MemoryPointer.new(:size_t)
  len_ptr = ::FFI::MemoryPointer.new(:size_t)
  begin
    records_ptr = Leptris::XML::FFI.leptris_sax_recorder_records(
      @handle, count_ptr)
    return self if records_ptr.null? || count_ptr.read_uint64.zero?

    arena_ptr = Leptris::XML::FFI.leptris_sax_recorder_arena(
      @handle, len_ptr)
    arena = arena_ptr.read_bytes(len_ptr.read_uint64)
    count = count_ptr.read_uint64

    # Two-level drain: ONE bulk read + kind-strip unpack decides
    # which records to touch, then each yielded event's fields are
    # read individually (get_uint32 at layout offsets) — an
    # unwanted record costs one strip read, an unused FIELD costs
    # nothing, and no per-chunk field array is ever built.
    records = records_ptr.read_bytes(count * RECORD_STRIDE)
    kinds = records.unpack(KIND_TEMPLATE_FOR[count])

    has_name = HAS_NAME
    has_text = HAS_TEXT
    drain = wanted.nil?
    i = 0
    while i < count
      code = kinds[i]
      base = i * RECORD_STRIDE
      if code == SAX_EVENT_CHARACTERS_CODE && drain
        # The dominant branch (text-heavy documents): no name, no
        # attrs, no per-kind gate lookups — kind, text, position.
        yield :characters, nil,
              slice(arena, records_ptr.get_uint32(base + TEXT_OFF),
                           records_ptr.get_uint32(base + TEXT_LEN)),
              nil,
              records_ptr.get_uint32(base + LINE_OFF),
              records_ptr.get_uint32(base + COLUMN_OFF)
      else
        kind = KIND_BY_CODE[code]
        if kind && (drain || wanted[kind])
          name = if has_name[kind]
                   slice(arena, records_ptr.get_uint32(base + NAME_OFF),
                                records_ptr.get_uint32(base + NAME_LEN))
                 end
          text = if has_text[kind]
                   slice(arena, records_ptr.get_uint32(base + TEXT_OFF),
                                records_ptr.get_uint32(base + TEXT_LEN))
                 end
          text = Leptris::XML::FFI.read_pi_data(text) if kind == :pi
          yield kind, name, text,
                attrs_from(arena, records_ptr.get_uint32(base + ATTRS_OFF),
                                  records_ptr.get_uint32(base + ATTR_COUNT)),
                records_ptr.get_uint32(base + LINE_OFF),
                records_ptr.get_uint32(base + COLUMN_OFF)
        end
      end
      i += 1
    end
  ensure
    count_ptr.free
    len_ptr.free
  end
  self
end

#feed(chunk, final: false) ⇒ Object

Feed one chunk; records/arena reset at feed entry, so drain after every feed. Returns self.



86
87
88
89
90
91
 
# File 'lib/leptris/xml/sax/recorder.rb', line 86

def feed(chunk, final: false)
  chunk = chunk.dup.force_encoding(Encoding::UTF_8)
  rc = Leptris::XML::FFI.leptris_sax_recorder_feed(
    @handle, chunk, chunk.bytesize, final ? 1 : 0)
  rc
end

#feed_stream(io, chunk_size = 65_536, kinds: nil, &block) ⇒ Object

Feed an IO in chunks, draining events after each chunk.



94
95
96
97
98
99
100
101
 
# File 'lib/leptris/xml/sax/recorder.rb', line 94

def feed_stream(io, chunk_size = 65_536, kinds: nil, &block)
  while (chunk = io.read(chunk_size))
    feed(chunk)
    each_event(*Array(kinds), &block)
  end
  feed("", final: true)
  each_event(*Array(kinds), &block)
end

#free ⇒ Object 



66
67
68
69
70
 
# File 'lib/leptris/xml/sax/recorder.rb', line 66

def free
  return if @handle.nil?
  Leptris::XML::FFI.leptris_sax_recorder_free(@handle)
  @handle = nil
end

#reset ⇒ Object

Resets a finished recorder for the next document (libleptris 1.9.10, upstream #594): fresh parser state, record/arena buffers retained — a one-document-per-parse loop reuses ONE recorder without new/free churn. Records restart empty.



76
77
78
79
80
81
82
 
# File 'lib/leptris/xml/sax/recorder.rb', line 76

def reset
  rc = Leptris::XML::FFI.leptris_sax_recorder_reset(@handle)
  if rc != 0
    raise Leptris::XML::Error, "leptris_sax_recorder_reset failed"
  end
  self
end