Module: Exwiw::Adapter::SqlBulkInsert

Included in:
MysqlAdapter, PostgresqlAdapter, SqliteAdapter
Defined in:
lib/exwiw/adapter/sql_bulk_insert.rb

Overview

Shared bulk-INSERT construction for the SQL adapters (mysql / postgresql / sqlite). They produce the same ‘INSERT INTO … VALUES (…),(…);` shape and differ only in the header’s identifier quoting (see #insert_header) and in #escape_value, so both the in-memory builder (#to_bulk_insert) and the bounded-memory streaming writer (#write_inserts) live here.

Each including adapter must provide two private methods:

- insert_header(table) -> the "INSERT INTO ... VALUES\n" prefix
- escape_value(value)  -> the SQL literal for one column value

Constant Summary collapse

STREAM_FLUSH_ROWS =

How many rows’ tuples to build-and-flush at a time when streaming. Bounds peak memory to this many tuples (plus their joined string) instead of the whole table’s INSERT string, while keeping each flush a single fast Array#map + Array#join (the same C-level path #to_bulk_insert uses) so it stays close to whole-string speed — far faster than a naive row-at-a-time IO#print (see script/bench_sql_dump.rb / docs/sql-dump-optimization-notes.md). Mirrors MongoDB’s default chunk size: bounded work per flush, but the SQL adapters still emit ONE statement (byte-identical to the un-chunked build).

2_000

Instance Method Summary collapse

Instance Method Details

#to_bulk_insert(results, table) ⇒ Object

Build the whole INSERT statement as a single String. Kept for callers that want the string form (and as the readable definition of the exact bytes #write_inserts streams).



28
29
30
31
 
# File 'lib/exwiw/adapter/sql_bulk_insert.rb', line 28

def to_bulk_insert(results, table)
  value_list = results.map { |row| insert_tuple(row) }
  "#{insert_header(table)}#{value_list.join(",\n")};"
end

#write_inserts(io, results, table, chunk_size) ⇒ Object

Stream the bulk INSERT(s) straight to ‘io` instead of materializing the whole statement string first. Byte-for-byte identical to writing #to_bulk_insert per chunk joined by “n” (verified by insert_output_snapshot_spec), but only one ~STREAM_FLUSH_BYTES buffer is resident at a time rather than the entire table’s INSERT string. Returns the number of statements written.



39
40
41
42
43
44
45
46
47
48
 
# File 'lib/exwiw/adapter/sql_bulk_insert.rb', line 39

def write_inserts(io, results, table, chunk_size)
  chunks = chunk_size ? results.each_slice(chunk_size) : [results]
  statement_count = 0
  chunks.each do |chunk_rows|
    io.print("\n") if statement_count.positive?
    stream_single_insert(io, chunk_rows, table)
    statement_count += 1
  end
  statement_count
end