Class: Exwiw::RowTransformer

Inherits:
Object
  • Object
show all
Defined in:
lib/exwiw/row_transformer.rb

Overview

Applies the Ruby-process-side masking modes — map and replace_with_fake_data — to the rows streamed out of a SQL adapter's #execute. Unlike replace_with/raw_sql, which compile into the SELECT and run in the database, these run in the exwiw process, so they can execute arbitrary Ruby (map) or derive deterministic fake values (fake data).

Built once per table (.build compiles the map procs, resolves seed column indexes, and pre-generates the fake-value pools), then #wrap decorates the adapter's StreamingResult: rows are transformed one at a time as the stream is drained, so the bounded-memory profile of the streaming dump is preserved. #size delegates to the underlying result, keeping the COPY path's upfront count and each_slice's allocation-hint COUNT identical to an unwrapped run.

SQL adapters only: .build returns nil for configs without a columns list (MongodbCollectionConfig has fields; rails-managed tables have no columns), and for tables where no column carries a Ruby-side mode.

Defined Under Namespace

Classes: Person, Row, TransformedResult

Constant Summary collapse

POOL_SIZE =

Fake values are picked from a pool of this many pre-generated values. Distinct seed values beyond the pool size reuse pool entries (fine for fake data — determinism, not uniqueness, is the contract; uniqueness- sensitive types compose a 64-bit token on top, see FAKE_TYPES).

10_000
POOL_RANDOM_SEED =

Fixed Random seed for pool generation, so a pool is deterministic for a given faker gem version + locale.

715_517
PERSON_POOL_SIZE =

The person pool (PERSON_TYPES) is sized independently of POOL_SIZE — at twice the size — so the coherent-identity name space is larger without disturbing the independent types' seed->value mappings. The ja pool is filled with distinct people (see .build_japanese_person_pool), so JapaneseNames must supply at least this many (surname, given) combinations.

POOL_SIZE * 2
PERSON_TYPES =

Person-family types: they share the per-locale person pool (built by .person_pool), and each extractor derives its value from the picked Person. Full-name ordering is locale-aware ("姓 名" for ja, "First Last" elsewhere). This is what keeps a single seed's name columns consistent.

{
  "human_name"      => ->(p, locale) { join_full_name(p.last_name, p.first_name, locale) },
  "last_name"       => ->(p, _locale) { p.last_name },
  "first_name"      => ->(p, _locale) { p.first_name },
  "human_name_kana" => ->(p, locale) { join_full_name(p.last_name_kana, p.first_name_kana, locale) },
  "last_name_kana"  => ->(p, _locale) { p.last_name_kana },
  "first_name_kana" => ->(p, _locale) { p.first_name_kana },
}.freeze
KANA_TYPES =

Person-family types whose value is a kana reading, only available for the ja locale (the person pool carries nil kana for other locales, so these are rejected at build time with a clear error).

%w[human_name_kana last_name_kana first_name_kana].freeze
FAKE_TYPES =

Independent (non-person) types. pool builds one candidate value (called POOL_SIZE times under a seeded Faker random). Types with compose are uniqueness-sensitive: the pooled base alone would collide under a unique index at scale, so the final value composes the base with a 64-bit hex token derived from the same seed digest (collision probability at 5M distinct seeds ≈ 7e-7).

{
  "phone_number" => { pool: -> { Faker::PhoneNumber.phone_number } },
  "address"      => { pool: -> { Faker::Address.full_address } },
  "company_name" => { pool: -> { Faker::Company.name } },
  "email"        => { pool: -> { Faker::Internet.username(specifier: 5..12) },
                      compose: ->(base, token) { "#{base}.#{token}@example.com" } },
  "username"     => { pool: -> { Faker::Internet.username(specifier: 5..12) },
                      compose: ->(base, token) { "#{base}_#{token}" } },
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(table) ⇒ RowTransformer

Returns a new instance of RowTransformer.



234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/exwiw/row_transformer.rb', line 234

def initialize(table)
  @table_name = table.name
  @name_to_index = {}
  table.columns.each_with_index { |column, index| @name_to_index[column.name] = index }
  @name_to_index.freeze
  @row_accessor = Row.new(@table_name, @name_to_index)

  needs_faker = table.columns.any? do |column|
    fd = column.replace_with_fake_data
    fd && self.class.type_needs_faker?(fd.type, fd.locale)
  end
  self.class.require_faker! if needs_faker

  @transforms = table.columns.each_with_index.filter_map do |column, index|
    if column.map
      [index, compile_map(column)]
    elsif column.replace_with_fake_data
      [index, compile_fake(column, index)]
    end
  end
  @replacement_buffer = Array.new(@transforms.size)
end

Class Method Details

.build(table) ⇒ Object

-> RowTransformer | nil. nil when the table carries no Ruby-side mode, so the Runner can skip wrapping entirely (the unused path stays byte-identical and cost-free).



143
144
145
146
147
148
149
150
 
# File 'lib/exwiw/row_transformer.rb', line 143

def self.build(table)
  return nil unless table.respond_to?(:columns)

  columns = table.columns
  return nil if columns.nil? || columns.none? { |c| c.map || c.replace_with_fake_data }

  new(table)
end

.build_fake_pool(type, locale) ⇒ Object 



177
178
179
180
181
182
183
184
185
186
187
 
# File 'lib/exwiw/row_transformer.rb', line 177

def self.build_fake_pool(type, locale)
  spec = FAKE_TYPES.fetch(type)
  previous_locale = Faker::Config.locale
  previous_random = Faker::Config.random
  Faker::Config.locale = locale if locale
  Faker::Config.random = Random.new(POOL_RANDOM_SEED)
  Array.new(POOL_SIZE) { spec[:pool].call.freeze }.freeze
ensure
  Faker::Config.locale = previous_locale
  Faker::Config.random = previous_random
end

.build_japanese_person_pool(random) ⇒ Object

Fill the ja pool with DISTINCT people: enumerate every (surname, given) combination, shuffle deterministically, and take PERSON_POOL_SIZE. Unlike sampling with replacement (which wastes ~37% of slots to duplicates), this gives PERSON_POOL_SIZE distinct identities — so JapaneseNames must supply at least that many combinations.



219
220
221
222
223
224
225
226
227
228
229
230
231
232
 
# File 'lib/exwiw/row_transformer.rb', line 219

def self.build_japanese_person_pool(random)
  surnames = JapaneseNames::SURNAMES
  given_names = JapaneseNames::GIVEN_NAMES
  combinations = surnames.size * given_names.size
  if combinations < PERSON_POOL_SIZE
    raise "JapaneseNames has #{combinations} (surname, given) combinations, " \
          "need at least PERSON_POOL_SIZE=#{PERSON_POOL_SIZE}"
  end

  people = surnames.flat_map do |last|
    given_names.map { |first| Person.new(last[0], first[0], last[1], first[1]).freeze }
  end
  people.shuffle(random: random).first(PERSON_POOL_SIZE).freeze
end

.build_person_pool(locale) ⇒ Object 



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
 
# File 'lib/exwiw/row_transformer.rb', line 198

def self.build_person_pool(locale)
  random = Random.new(POOL_RANDOM_SEED)
  return build_japanese_person_pool(random) if locale.to_s == "ja"

  previous_locale = Faker::Config.locale
  previous_random = Faker::Config.random
  Faker::Config.locale = locale if locale
  Faker::Config.random = random
  Array.new(PERSON_POOL_SIZE) { Person.new(Faker::Name.last_name, Faker::Name.first_name, nil, nil).freeze }.freeze
ensure
  unless locale.to_s == "ja"
    Faker::Config.locale = previous_locale
    Faker::Config.random = previous_random
  end
end

.fake_pool(type, locale) ⇒ Object

Pools are memoized per (type, locale): every column sharing a type+locale sees the same pool, which is what makes equal seed values map to equal fake values across tables and runs.



172
173
174
175
 
# File 'lib/exwiw/row_transformer.rb', line 172

def self.fake_pool(type, locale)
  @fake_pools ||= {}
  @fake_pools[[type, locale]] ||= build_fake_pool(type, locale)
end

.join_full_name(last, first, locale) ⇒ Object

"姓 名" for Japanese, "First Last" otherwise. A nil part (e.g. missing kana) collapses so the result never has a dangling separator.



87
88
89
90
 
# File 'lib/exwiw/row_transformer.rb', line 87

def self.join_full_name(last, first, locale)
  parts = locale.to_s == "ja" ? [last, first] : [first, last]
  parts.compact.join(" ")
end

.person_pool(locale) ⇒ Object

One pool of coherent Person records per locale, shared by every person-family type. Memoized so equal seeds map to equal people across tables and runs. ja is built from exwiw's bundled (kanji, kana) dataset so kana matches kanji; other locales use faker (no kana).



193
194
195
196
 
# File 'lib/exwiw/row_transformer.rb', line 193

def self.person_pool(locale)
  @person_pools ||= {}
  @person_pools[locale] ||= build_person_pool(locale)
end

.require_faker! ⇒ Object 



152
153
154
155
156
157
158
 
# File 'lib/exwiw/row_transformer.rb', line 152

def self.require_faker!
  require "faker"
rescue LoadError
  raise LoadError,
        "replace_with_fake_data requires the faker gem. " \
        "Add `gem \"faker\"` to your Gemfile (it is not a runtime dependency of exwiw)."
end

.type_needs_faker?(type, locale) ⇒ Boolean

Whether a (type, locale) draws any value from faker. The ja person pool is built entirely from exwiw's bundled dataset, so a config that only uses ja person types needs no faker at all.

Returns:

  • (Boolean) 


163
164
165
166
167
 
# File 'lib/exwiw/row_transformer.rb', line 163

def self.type_needs_faker?(type, locale)
  return true if FAKE_TYPES.key?(type)

  PERSON_TYPES.key?(type) && locale.to_s != "ja"
end

Instance Method Details

#transform(row) ⇒ Object

Replacements are all computed from the original row before any is written back, so a map proc / fake seed always reads the pre-transform (post-SQL-masking) value regardless of column order. Rows are mutated in place when possible (each cursor yields a fresh Array per row), but sqlite3's Statement#each yields frozen rows — those are duped first.



266
267
268
269
270
271
272
273
274
275
 
# File 'lib/exwiw/row_transformer.rb', line 266

def transform(row)
  @transforms.each_with_index do |(_, callable), k|
    @replacement_buffer[k] = callable.call(row)
  end
  row = row.dup if row.frozen?
  @transforms.each_with_index do |(index, _), k|
    row[index] = @replacement_buffer[k]
  end
  row
end

#wrap(results) ⇒ Object 



257
258
259
 
# File 'lib/exwiw/row_transformer.rb', line 257

def wrap(results)
  TransformedResult.new(results, self)
end