Class: StructuredDataToSql::Json::RecordStreamer

Inherits:
Object
  • Object
show all
Defined in:
lib/structured_data_to_sql/json/record_streamer.rb

Overview

Streams the records out of one JSON document without materializing the whole document: only the envelope (everything outside the records array) and one record at a time are held in memory.

Records-array detection, first match wins:

1. the array at the end of the explicit dot-notation records_path
 (e.g. "data.community.roles.edges"); error if absent
2. the array under the top-level "data" key
3. the sole top-level array value (buffered into the envelope first,
 so only suitable for modest non-"data" files; use records_path for
 huge ones)
4. none: the whole document is a single record

NDJSON (newline-delimited JSON) support:

When ndjson: true is passed, or auto-detection determines the stream
contains multiple top-level JSON documents (one per line), each line is
parsed independently and records from all lines are yielded in order.
All lines must share the same structure (same records_path). The
envelope is taken from the first non-empty line; subsequent lines'
envelopes are ignored (they are structurally identical page wrappers).

Defined Under Namespace

Classes: Handler, Result

Constant Summary collapse

NDJSON_PROBE_LINE_BYTES =

Maximum bytes read for each line considered during NDJSON probing.

16 * 1024 * 1024

Instance Method Summary collapse

Constructor Details

#initialize(records_path: nil) ⇒ RecordStreamer

Returns a new instance of RecordStreamer.



42
43
44
 
# File 'lib/structured_data_to_sql/json/record_streamer.rb', line 42

def initialize(records_path: nil)
  @records_path = records_path
end

Instance Method Details

#each_record(io, recover: false, ndjson: :auto, &block) ⇒ Object

Yields each record (a plain Hash/Array/scalar tree) with its 0-based ordinal. Returns a Result with the envelope hash (records excluded).

With recover: true, a parse failure inside the records array returns a truncated Result covering the records that completed before the failure (each yielded record is a fully parsed tree, so everything already yielded is intact); the partial tail record is discarded. Failures before the first complete record still raise.

With ndjson: true, each line of io is treated as a separate JSON document. Auto-detection (ndjson: :auto, the default) probes the first two non-empty lines; set ndjson: false to disable auto-detection.



58
59
60
61
62
63
64
 
# File 'lib/structured_data_to_sql/json/record_streamer.rb', line 58

def each_record(io, recover: false, ndjson: :auto, &block)
  if ndjson == true || (ndjson == :auto && ndjson_stream?(io))
    each_record_ndjson(io, recover: recover, &block)
  else
    each_record_single(io, recover: recover, &block)
  end
end