Class: Exwiw::RowTransformer

Inherits:
Object
  • Object
show all
Defined in:
lib/exwiw/row_transformer.rb

Overview

Applies the Ruby-process-side masking modes — map and replace_with_fake_data — to the rows streamed out of a SQL adapter's #execute. Unlike replace_with/raw_sql, which compile into the SELECT and run in the database, these run in the exwiw process, so they can execute arbitrary Ruby (map) or derive deterministic fake values (fake data).

Built once per table (.build compiles the map procs, resolves seed column indexes, and pre-generates the fake-value pools), then #wrap decorates the adapter's StreamingResult: rows are transformed one at a time as the stream is drained, so the bounded-memory profile of the streaming dump is preserved. #size delegates to the underlying result, keeping the COPY path's upfront count and each_slice's allocation-hint COUNT identical to an unwrapped run.

SQL adapters only: .build returns nil for configs without a columns list (MongodbCollectionConfig has fields; rails-managed tables have no columns), and for tables where no column carries a Ruby-side mode.

Defined Under Namespace

Classes: Person, Row, TransformedResult

Constant Summary collapse

POOL_SIZE =

Fake values are picked from a pool of this many pre-generated values. Distinct seed values beyond the pool size reuse pool entries (fine for fake data — determinism, not uniqueness, is the contract; uniqueness- sensitive types compose a 64-bit token on top, see FAKE_TYPES).

10_000
POOL_RANDOM_SEED =

Fixed Random seed for pool generation, so a pool is deterministic for a given faker gem version + locale.

715_517
PERSON_POOL_SIZE =

The person pool (PERSON_TYPES) is sized independently of POOL_SIZE — at twice the size — so the coherent-identity name space is larger without disturbing the independent types' seed->value mappings. The ja pool is filled with distinct people (see .build_japanese_person_pool), so JapaneseNames must supply at least this many (surname, given) combinations.

POOL_SIZE * 2
PERSON_TYPES =

Person-family types: they share the per-locale person pool (built by .person_pool), and each extractor derives its value from the picked Person. Full-name ordering is locale-aware ("姓 名" for ja, "First Last" elsewhere). This is what keeps a single seed's name columns consistent.

{
  "human_name"      => ->(p, locale) { join_full_name(p.last_name, p.first_name, locale) },
  "last_name"       => ->(p, _locale) { p.last_name },
  "first_name"      => ->(p, _locale) { p.first_name },
  "human_name_kana" => ->(p, locale) { join_full_name(p.last_name_kana, p.first_name_kana, locale) },
  "last_name_kana"  => ->(p, _locale) { p.last_name_kana },
  "first_name_kana" => ->(p, _locale) { p.first_name_kana },
}.freeze
KANA_TYPES =

Person-family types whose value is a kana reading, only available for the ja locale (the person pool carries nil kana for other locales, so these are rejected at build time with a clear error).

%w[human_name_kana last_name_kana first_name_kana].freeze
FAKE_TYPES =

Independent (non-person) types. pool builds one candidate value (called POOL_SIZE times under a seeded Faker random). Types with compose are uniqueness-sensitive: the pooled base alone would collide under a unique index at scale, so the final value composes the base with a 64-bit hex token derived from the same seed digest (collision probability at 5M distinct seeds ≈ 7e-7).

{
  "phone_number" => { pool: -> { Faker::PhoneNumber.phone_number } },
  "address"      => { pool: -> { Faker::Address.full_address } },
  "company_name" => { pool: -> { Faker::Company.name } },
  "email"        => { pool: -> { Faker::Internet.username(specifier: 5..12) },
                      compose: ->(base, token) { "#{base}.#{token}@example.com" } },
  "username"     => { pool: -> { Faker::Internet.username(specifier: 5..12) },
                      compose: ->(base, token) { "#{base}_#{token}" } },
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(table) ⇒ RowTransformer

Returns a new instance of RowTransformer.



293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
 
# File 'lib/exwiw/row_transformer.rb', line 293

def initialize(table)
  @table_name = table.name
  @name_to_index = {}
  table.columns.each_with_index { |column, index| @name_to_index[column.name] = index }
  @name_to_index.freeze
  @row_accessor = Row.new(@table_name, @name_to_index)

  needs_faker = table.columns.any? do |column|
    fd = column.replace_with_fake_data
    fd && self.class.type_needs_faker?(fd.type, fd.locale)
  end
  self.class.require_faker! if needs_faker

  @transforms = table.columns.each_with_index.filter_map do |column, index|
    if column.map
      [index, compile_map(column)]
    elsif column.replace_with_fake_data
      [index, compile_fake(column, index)]
    end
  end
  @replacement_buffer = Array.new(@transforms.size)
end

Class Method Details

.build(table) ⇒ Object

-> RowTransformer | nil. nil when the table carries no Ruby-side mode, so the Runner can skip wrapping entirely (the unused path stays byte-identical and cost-free).



143
144
145
146
147
148
149
150
 
# File 'lib/exwiw/row_transformer.rb', line 143

def self.build(table)
  return nil unless table.respond_to?(:columns)

  columns = table.columns
  return nil if columns.nil? || columns.none? { |c| c.map || c.replace_with_fake_data }

  new(table)
end

.build_fake_pool(type, locale) ⇒ Object 



177
178
179
180
181
182
183
184
185
186
187
 
# File 'lib/exwiw/row_transformer.rb', line 177

def self.build_fake_pool(type, locale)
  spec = FAKE_TYPES.fetch(type)
  previous_locale = Faker::Config.locale
  previous_random = Faker::Config.random
  Faker::Config.locale = locale if locale
  Faker::Config.random = Random.new(POOL_RANDOM_SEED)
  Array.new(POOL_SIZE) { spec[:pool].call.freeze }.freeze
ensure
  Faker::Config.locale = previous_locale
  Faker::Config.random = previous_random
end

.build_independent_deriver(fake_data) ⇒ Object

Independent-type deriver: pick a value from its own (type, locale) pool, composing a 64-bit token for uniqueness-sensitive types (email/username).



267
268
269
270
271
272
273
274
275
276
 
# File 'lib/exwiw/row_transformer.rb', line 267

def self.build_independent_deriver(fake_data)
  pool = fake_pool(fake_data.type, fake_data.locale)
  compose = FAKE_TYPES.fetch(fake_data.type)[:compose]

  lambda do |seed_value|
    digest = Digest::SHA256.digest(seed_value.to_s)
    base = pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
    compose ? compose.call(base, digest[8, 8].unpack1("H*")) : base
  end
end

.build_japanese_person_pool(random) ⇒ Object 



278
279
280
281
282
283
284
285
286
287
288
289
290
291
 
# File 'lib/exwiw/row_transformer.rb', line 278

def self.build_japanese_person_pool(random)
  surnames = JapaneseNames::SURNAMES
  given_names = JapaneseNames::GIVEN_NAMES
  combinations = surnames.size * given_names.size
  if combinations < PERSON_POOL_SIZE
    raise "JapaneseNames has #{combinations} (surname, given) combinations, " \
          "need at least PERSON_POOL_SIZE=#{PERSON_POOL_SIZE}"
  end

  people = surnames.flat_map do |last|
    given_names.map { |first| Person.new(last[0], first[0], last[1], first[1]).freeze }
  end
  people.shuffle(random: random).first(PERSON_POOL_SIZE).freeze
end

.build_person_deriver(fake_data, error_context) ⇒ Object

Person-family deriver: pick a coherent Person from the per-locale pool. The digest index is shared with every other person column/field, so one seed's name values all belong to the same person.



246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
 
# File 'lib/exwiw/row_transformer.rb', line 246

def self.build_person_deriver(fake_data, error_context)
  type = fake_data.type
  locale = fake_data.locale
  pool = person_pool(locale)
  extractor = PERSON_TYPES.fetch(type)

  if KANA_TYPES.include?(type) && pool.first.last_name_kana.nil?
    raise ArgumentError,
          "replace_with_fake_data for #{error_context}: type '#{type}' needs kana readings, " \
          "which are only available with locale: ja (got locale: #{locale.inspect})"
  end

  lambda do |seed_value|
    digest = Digest::SHA256.digest(seed_value.to_s)
    person = pool[digest[0, 8].unpack1("Q>") % PERSON_POOL_SIZE]
    extractor.call(person, locale)
  end
end

.build_person_pool(locale) ⇒ Object 



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
 
# File 'lib/exwiw/row_transformer.rb', line 198

def self.build_person_pool(locale)
  random = Random.new(POOL_RANDOM_SEED)
  return build_japanese_person_pool(random) if locale.to_s == "ja"

  previous_locale = Faker::Config.locale
  previous_random = Faker::Config.random
  Faker::Config.locale = locale if locale
  Faker::Config.random = random
  Array.new(PERSON_POOL_SIZE) { Person.new(Faker::Name.last_name, Faker::Name.first_name, nil, nil).freeze }.freeze
ensure
  unless locale.to_s == "ja"
    Faker::Config.locale = previous_locale
    Faker::Config.random = previous_random
  end
end

.build_value_deriver(fake_data, error_context) ⇒ Object

Fill the ja pool with DISTINCT people: enumerate every (surname, given) combination, shuffle deterministically, and take PERSON_POOL_SIZE. Unlike sampling with replacement (which wastes ~37% of slots to duplicates), this gives PERSON_POOL_SIZE distinct identities — so JapaneseNames must supply at least that many combinations. Compile a FakeData config into a callable seed_value -> fake value, requiring faker if the (type, locale) needs it. Shared by the SQL RowTransformer (per-row array) and the MongoDB mask plan (per-document hash): both derive byte-identical fake values from the same seed, keeping replace_with_fake_data consistent across adapters. The caller owns NULL preservation of the target (a nil seed still hashes "", deterministically, exactly like the SQL path). error_context names the offending column/field in raised messages.



227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
 
# File 'lib/exwiw/row_transformer.rb', line 227

def self.build_value_deriver(fake_data, error_context)
  type = fake_data.type
  unless PERSON_TYPES.key?(type) || FAKE_TYPES.key?(type)
    raise ArgumentError,
          "replace_with_fake_data for #{error_context}: unknown type '#{type}'. " \
          "Supported: #{(PERSON_TYPES.keys + FAKE_TYPES.keys).join(', ')}"
  end
  require_faker! if type_needs_faker?(type, fake_data.locale)

  if PERSON_TYPES.key?(type)
    build_person_deriver(fake_data, error_context)
  else
    build_independent_deriver(fake_data)
  end
end

.fake_pool(type, locale) ⇒ Object

Pools are memoized per (type, locale): every column sharing a type+locale sees the same pool, which is what makes equal seed values map to equal fake values across tables and runs.



172
173
174
175
 
# File 'lib/exwiw/row_transformer.rb', line 172

def self.fake_pool(type, locale)
  @fake_pools ||= {}
  @fake_pools[[type, locale]] ||= build_fake_pool(type, locale)
end

.join_full_name(last, first, locale) ⇒ Object

"姓 名" for Japanese, "First Last" otherwise. A nil part (e.g. missing kana) collapses so the result never has a dangling separator.



87
88
89
90
 
# File 'lib/exwiw/row_transformer.rb', line 87

def self.join_full_name(last, first, locale)
  parts = locale.to_s == "ja" ? [last, first] : [first, last]
  parts.compact.join(" ")
end

.person_pool(locale) ⇒ Object

One pool of coherent Person records per locale, shared by every person-family type. Memoized so equal seeds map to equal people across tables and runs. ja is built from exwiw's bundled (kanji, kana) dataset so kana matches kanji; other locales use faker (no kana).



193
194
195
196
 
# File 'lib/exwiw/row_transformer.rb', line 193

def self.person_pool(locale)
  @person_pools ||= {}
  @person_pools[locale] ||= build_person_pool(locale)
end

.require_faker! ⇒ Object 



152
153
154
155
156
157
158
 
# File 'lib/exwiw/row_transformer.rb', line 152

def self.require_faker!
  require "faker"
rescue LoadError
  raise LoadError,
        "replace_with_fake_data requires the faker gem. " \
        "Add `gem \"faker\"` to your Gemfile (it is not a runtime dependency of exwiw)."
end

.type_needs_faker?(type, locale) ⇒ Boolean

Whether a (type, locale) draws any value from faker. The ja person pool is built entirely from exwiw's bundled dataset, so a config that only uses ja person types needs no faker at all.

Returns:

  • (Boolean) 


163
164
165
166
167
 
# File 'lib/exwiw/row_transformer.rb', line 163

def self.type_needs_faker?(type, locale)
  return true if FAKE_TYPES.key?(type)

  PERSON_TYPES.key?(type) && locale.to_s != "ja"
end

Instance Method Details

#transform(row) ⇒ Object

Replacements are all computed from the original row before any is written back, so a map proc / fake seed always reads the pre-transform (post-SQL-masking) value regardless of column order. Rows are mutated in place when possible (each cursor yields a fresh Array per row), but sqlite3's Statement#each yields frozen rows — those are duped first.



325
326
327
328
329
330
331
332
333
334
 
# File 'lib/exwiw/row_transformer.rb', line 325

def transform(row)
  @transforms.each_with_index do |(_, callable), k|
    @replacement_buffer[k] = callable.call(row)
  end
  row = row.dup if row.frozen?
  @transforms.each_with_index do |(index, _), k|
    row[index] = @replacement_buffer[k]
  end
  row
end

#wrap(results) ⇒ Object 



316
317
318
 
# File 'lib/exwiw/row_transformer.rb', line 316

def wrap(results)
  TransformedResult.new(results, self)
end