Class: Synthra::NativeEngine

Inherits:
Object
  • Object
show all
Defined in:
lib/synthra/native_engine.rb

Overview

High-performance native engine using Rust + fake-rs

Provides 100-250x speedup over pure Ruby implementation by using a native Rust extension with the fake-rs library.

Examples:

Check availability

if Synthra::NativeEngine.available?
  puts "Native engine is available!"
end

Basic usage

schema = Synthra.parse("User:\n  id: uuid\n  name: name")
engine = Synthra::NativeEngine.new(schema)
records = engine.generate_many(1000)

Generate to file

engine.generate_to_file(10_000_000, "users.jsonl", format: "jsonl")

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(schema, locale: "en", threads: nil) ⇒ NativeEngine

Initialize native engine with a schema

Parameters:

  • schema (Synthra::Schema) —

    the schema to generate from

  • locale (String) (defaults to: "en") —

    locale for fake data (en, fr_fr, de_de, ja_jp, etc.)

  • threads (Integer, nil) (defaults to: nil) —

    number of threads (nil/0 = use all cores)

Raises:

  • (LoadError) —

    if native extension is not available



136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/synthra/native_engine.rb', line 136

def initialize(schema, locale: "en", threads: nil)
  unless self.class.available?
    raise LoadError, "Native extension not available. " \
                     "Install Rust and run `bundle exec rake compile`"
  end

  @schema = schema
  @locale = locale.to_s
  @threads = threads
  @field_specs = build_field_specs
end

Instance Attribute Details

#locale ⇒ String (readonly)

Returns the locale for data generation.

Returns:

  • (String) —

    the locale for data generation



128
129
130
 
# File 'lib/synthra/native_engine.rb', line 128

def locale
  @locale
end

#schema ⇒ Synthra::Schema (readonly)

Returns the schema being used.

Returns:



125
126
127
 
# File 'lib/synthra/native_engine.rb', line 125

def schema
  @schema
end

Class Method Details

.available? ⇒ Boolean

Check if native extension is available

Returns:

  • (Boolean) —

    true if the native extension is loaded and functional



27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/synthra/native_engine.rb', line 27

def available?
  return @available if defined?(@available)

  @available = begin
    load_native_extension
    defined?(Synthra::Native) &&
      Native.respond_to?(:loaded?) &&
      Native.loaded?
  rescue LoadError, NameError
    false
  end
end

.available_locales ⇒ Array<String>

Get list of locales supported by native engine

Returns:

  • (Array<String>) —

    list of locale codes



52
53
54
55
56
 
# File 'lib/synthra/native_engine.rb', line 52

def available_locales
  return [] unless available?

  Native.available_locales
end

.available_types ⇒ Array<String>

Get list of types supported by native engine

Returns:

  • (Array<String>) —

    list of type names



43
44
45
46
47
 
# File 'lib/synthra/native_engine.rb', line 43

def available_types
  return [] unless available?

  Native.available_types
end

.load_native_extension ⇒ Object (private) 



104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/synthra/native_engine.rb', line 104

def load_native_extension
  require "synthra/synthra_native"
rescue LoadError => e
  # Check if it's a missing extension or missing dependencies
  if e.message.include?("cannot load such file")
    raise LoadError, <<~MSG
      Native extension not compiled. To enable native performance:

      1. Install Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
      2. Install rb_sys: gem install rb_sys
      3. Compile extension: bundle exec rake compile

      Original error: #{e.message}
    MSG
  else
    raise
  end
end

.performance_info ⇒ String?

Get performance information

Returns:

  • (String, nil) —

    performance info or nil if not available



70
71
72
73
74
 
# File 'lib/synthra/native_engine.rb', line 70

def performance_info
  return nil unless available?

  Native.performance_info
end

.set_threads(num_threads) ⇒ Integer

Note:

Alternatively, set RAYON_NUM_THREADS env var before loading: RAYON_NUM_THREADS=4 ruby my_script.rb

Set the number of threads to use for parallel generation Must be called before any generation operations.

Examples:

Limit to 4 threads

Synthra::NativeEngine.set_threads(4)

Parameters:

  • num_threads (Integer) —

    number of threads (0 = use all cores)

Returns:

  • (Integer) —

    actual number of threads configured



96
97
98
99
100
 
# File 'lib/synthra/native_engine.rb', line 96

def set_threads(num_threads)
  return nil unless available?

  Native.set_threads(num_threads)
end

.thread_count ⇒ Integer?

Get the current number of threads used for parallel generation

Returns:

  • (Integer, nil) —

    thread count or nil if not available



79
80
81
82
83
 
# File 'lib/synthra/native_engine.rb', line 79

def thread_count
  return nil unless available?

  Native.thread_count
end

.version ⇒ String?

Get native extension version

Returns:

  • (String, nil) —

    version string or nil if not available



61
62
63
64
65
 
# File 'lib/synthra/native_engine.rb', line 61

def version
  return nil unless available?

  Native.version
end

Instance Method Details

#build_args_json(field) ⇒ Object (private)

Build args JSON for a field



264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
 
# File 'lib/synthra/native_engine.rb', line 264

def build_args_json(field)
  args = field.type_args || {}

  # Handle enum values
  if field.type_name.to_s == "enum" && args[:values]
    args = args.merge(values: args[:values].map { |v| extract_enum_value(v) })
  end

  # Handle const value
  if field.type_name.to_s == "const"
    args = { value: args[:value] || args.values.first }
  end

  # Handle number ranges
  if args[:range]
    range = args[:range]
    args = args.merge(min: range.begin, max: range.end)
  end

  args.to_json
end

#build_field_specs ⇒ Object (private)

Build field specs for native engine Format: [name, type_name, args_json, optional, nullable]



234
235
236
237
238
239
240
241
242
243
244
 
# File 'lib/synthra/native_engine.rb', line 234

def build_field_specs
  @schema.fields.map do |field|
    [
      field.name.to_s,
      map_type_name(field.type_name),
      build_args_json(field),
      field.optional?,
      field.nullable?
    ]
  end
end

#extract_enum_value(value) ⇒ Object (private)

Extract the string value from an enum value (handles AST nodes and hashes)



287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
 
# File 'lib/synthra/native_engine.rb', line 287

def extract_enum_value(value)
  case value
  when Hash
    value[:value] || value["value"]
  when String, Symbol
    value.to_s
  else
    # Handle AST nodes like EnumValueNode
    if value.respond_to?(:value)
      value.value.to_s
    else
      value.to_s
    end
  end
end

#generate(seed: nil) ⇒ Hash

Generate a single record

Examples:

record = engine.generate(seed: 42)

Parameters:

  • seed (Integer, nil) (defaults to: nil) —

    random seed for reproducibility

Returns:

  • (Hash) —

    generated record



171
172
173
 
# File 'lib/synthra/native_engine.rb', line 171

def generate(seed: nil)
  Native.generate_one(@field_specs, seed, @locale)
end

#generate_many(count, seed: nil, threads: nil) ⇒ Array<Hash>

Generate multiple records

Examples:

Generate with default threads

engine.generate_many(1000, seed: 42)

Generate with limited threads

engine.generate_many(1000, seed: 42, threads: 4)

Parameters:

  • count (Integer) —

    number of records to generate

  • seed (Integer, nil) (defaults to: nil) —

    random seed for reproducibility

  • threads (Integer, nil) (defaults to: nil) —

    number of threads (nil/0 = use all cores)

Returns:

  • (Array<Hash>) —

    array of generated records



160
161
162
 
# File 'lib/synthra/native_engine.rb', line 160

def generate_many(count, seed: nil, threads: nil)
  Native.generate(@field_specs, count, seed, @locale, threads || @threads)
end

#generate_to_file(count, path, seed: nil, format: "jsonl", threads: nil) ⇒ Integer Also known as: to_file

Generate records directly to file (most efficient for large datasets)

Examples:

Generate JSONL (one JSON per line)

engine.generate_to_file(10_000_000, "users.jsonl", format: "jsonl")

Generate CSV

engine.generate_to_file(1_000_000, "users.csv", format: "csv")

Generate with limited threads

engine.generate_to_file(10_000_000, "users.jsonl", threads: 4)

Parameters:

  • count (Integer) —

    number of records to generate

  • path (String) —

    output file path

  • seed (Integer, nil) (defaults to: nil) —

    random seed for reproducibility

  • format (String) (defaults to: "jsonl") —

    output format: "jsonl", "json", "csv"

  • threads (Integer, nil) (defaults to: nil) —

    number of threads (nil/0 = use all cores)

Returns:

  • (Integer) —

    number of records written



192
193
194
 
# File 'lib/synthra/native_engine.rb', line 192

def generate_to_file(count, path, seed: nil, format: "jsonl", threads: nil)
  Native.generate_to_file(@field_specs, count, path.to_s, seed, @locale, format.to_s, threads || @threads)
end

#map_type_name(type_name) ⇒ Object (private)

Map DSL type names to native engine type names



247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
 
# File 'lib/synthra/native_engine.rb', line 247

def map_type_name(type_name)
  type_str = type_name.to_s

  # Handle type aliases
  case type_str
  when "full_name" then "name"
  when "phone_number" then "phone"
  when "street_address" then "street"
  when "zip_code" then "postal_code"
  when "int", "integer" then "number"
  when "bool" then "boolean"
  when "timestamp" then "datetime"
  else type_str
  end
end

#stream(count, seed: nil) {|Hash| ... } ⇒ Enumerator

Stream records (yields each record)

Examples:

engine.stream(1000) do |record|
  process(record)
end

Parameters:

  • count (Integer) —

    number of records to generate

  • seed (Integer, nil) (defaults to: nil) —

    random seed for reproducibility

Yields:

  • (Hash) —

    each generated record

Returns:

  • (Enumerator) —

    if no block given



210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
 
# File 'lib/synthra/native_engine.rb', line 210

def stream(count, seed: nil, &block)
  return enum_for(:stream, count, seed: seed) unless block_given?

  # For now, generate in batches of 10K for memory efficiency
  batch_size = 10_000
  remaining = count
  offset = 0

  while remaining > 0
    current_batch = [remaining, batch_size].min
    batch_seed = seed ? seed + offset : nil

    records = Native.generate(@field_specs, current_batch, batch_seed, @locale, @threads)
    records.each(&block)

    remaining -= current_batch
    offset += current_batch
  end
end