Native Rust Engine

The FakeDataDSL Native Engine provides ultra-high-performance data generation using Rust and the fake-rs library. It achieves 3-4 million records per second, which is 100-250x faster than the pure Ruby implementation.

Features

  • ⚡ 3-4 million records/sec generation speed
  • 🌍 Multi-locale support (English, French, German, Japanese, Chinese, etc.)
  • 🔧 Configurable thread count (default: 2 threads)
  • 📁 Direct file output (JSONL, JSON, CSV)
  • 🎲 Deterministic seeding for reproducible results
  • 💾 Memory efficient for large datasets

Installation

Prerequisites

  1. Rust (1.70+): Install via rustup

    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

  2. rb_sys gem: For Ruby-Rust integration

    gem install rb_sys

Compile the Extension

cd fake_data_dsl
bundle exec rake compile

Verify Installation

require 'fake_data_dsl'

if FakeDataDSL::NativeEngine.available?
  puts "Native engine available!"
  puts FakeDataDSL::NativeEngine.performance_info
else
  puts "Native engine not available - using Ruby engine"
end

Usage

Basic Generation

require 'fake_data_dsl'

schema = FakeDataDSL.parse(<<~DSL)
  User:
    id: uuid
    name: name
    email: email
    age: number(18..80)
    active: boolean
DSL

# Generate using native engine
users = schema.generate_many(100_000, engine: :native, seed: 42)

Generate to File (Fastest)

For maximum performance with large datasets, generate directly to file:

# Generate 10 million records to JSONL (one JSON per line)
schema.generate_to_file(10_000_000, "users.jsonl", format: "jsonl")

# Generate CSV
schema.generate_to_file(1_000_000, "users.csv", format: "csv")

# Generate JSON array
schema.generate_to_file(100_000, "users.json", format: "json")

Thread Control

By default, the native engine uses 2 threads to leave resources for other work. You can customize this:

# Use 4 threads
schema.generate_to_file(10_000_000, "users.jsonl", threads: 4)

# Use all available cores (fastest for single job)
schema.generate_to_file(10_000_000, "users.jsonl", threads: 0)

# Use 1 thread (minimal resource usage)
schema.generate_to_file(10_000_000, "users.jsonl", threads: 1)
Threads Use Case
nil (default) 2 threads - balanced for general use
0 All cores - maximum speed for single job
1 Minimal - when running many parallel jobs
4-8 Good balance for multi-process environments

Locale Support

The native engine supports multiple locales for authentic names:

# English (uses fast pre-generated pools)
schema.generate_to_file(1_000_000, "users_en.jsonl", locale: "en")

# French
schema.generate_to_file(1_000_000, "users_fr.jsonl", locale: "fr_fr")

# German
schema.generate_to_file(1_000_000, "users_de.jsonl", locale: "de_de")

# Japanese
schema.generate_to_file(1_000_000, "users_ja.jsonl", locale: "ja_jp")

# Chinese (Simplified)
schema.generate_to_file(1_000_000, "users_zh.jsonl", locale: "zh_cn")

Supported Locales:

  • en - English (fast mode)
  • fr_fr - French
  • de_de - German
  • ja_jp - Japanese
  • zh_cn - Chinese (Simplified)
  • zh_tw - Chinese (Traditional)
  • pt_br - Portuguese (Brazil)
  • it_it - Italian
  • ar_sa - Arabic

Using NativeEngine Class Directly

For more control, use the NativeEngine class directly:

# Create engine with options
engine = FakeDataDSL::NativeEngine.new(
  schema,
  locale: "en",
  threads: 4
)

# Generate records
records = engine.generate_many(100_000, seed: 42)

# Generate to file
engine.generate_to_file(10_000_000, "users.jsonl", format: "jsonl")

# Stream records
engine.stream(1_000_000) do |record|
  process(record)
end

Performance Benchmarks

Records Time Rate
1M 0.3s ~3.5M/sec
5M 1.5s ~3.3M/sec
10M 2.7s ~3.7M/sec
20M 5.3s ~3.8M/sec

Comparison: Ruby vs Native

Engine 10M Records Rate
Pure Ruby ~10 minutes ~17K/sec
Native (2 threads) ~5 seconds ~2M/sec
Native (all cores) ~2.7 seconds ~3.7M/sec

Supported Types

The native engine supports these types:

Personal

  • name, full_name, first_name, last_name, title

Internet

  • email, username, password, url
  • ip, ipv4, ipv6, mac_address, user_agent, domain

Address

  • city, country, country_code, state
  • street, street_address, postal_code, zip_code
  • latitude, longitude

Company

  • company, company_name, catch_phrase, buzzword, industry

Text

  • word, words, sentence, sentences, paragraph, paragraphs, text

Phone

  • phone, phone_number, cell_phone

Date/Time

  • date, time, datetime, timestamp, past_date, future_date

Identifiers

  • uuid, uuid_v4, uuid_v1, id_sequence, ulid

Primitives

  • number, integer, float, boolean, bool

Color

  • hex_color, rgb_color

Special

  • enum, const, array

Environment Variables

You can also configure threads via environment variable:

# Set before running Ruby
RAYON_NUM_THREADS=4 ruby my_script.rb

Troubleshooting

Native engine not available

Native extension not compiled. To enable native performance:
1. Install Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
2. Install rb_sys: gem install rb_sys
3. Compile extension: bundle exec rake compile

Compilation errors

Make sure you have:

  • Rust 1.70+ installed
  • Clang/LLVM for linking
  • rb_sys gem installed
# Check Rust version
rustc --version

# Reinstall rb_sys
gem install rb_sys

# Clean and recompile
cd fake_data_dsl
rm -rf ext/fake_data_dsl_native/target
bundle exec rake compile

Performance not as expected

  1. Check thread count: Default is 2 threads. Use threads: 0 for max speed.
  2. Use file output: generate_to_file is faster than generate_many for large datasets.
  3. English locale: English uses fast pre-generated pools; other locales use full fake-rs.

API Reference

FakeDataDSL::NativeEngine

Class Methods

NativeEngine.available?        # Check if native extension is loaded
NativeEngine.version           # Get extension version
NativeEngine.performance_info  # Get performance information
NativeEngine.thread_count      # Get current thread count
NativeEngine.set_threads(n)    # Set thread count (call early)
NativeEngine.available_types   # List supported types
NativeEngine.available_locales # List supported locales

Instance Methods

engine = NativeEngine.new(schema, locale: "en", threads: 2)

engine.generate(seed: nil)                    # Single record
engine.generate_many(count, seed: nil, threads: nil)  # Multiple records
engine.generate_to_file(count, path, seed: nil, format: "jsonl", threads: nil)
engine.stream(count, seed: nil) { |record| }  # Stream records

Schema Methods

schema.generate_many(count, engine: :native, locale: "en", threads: 2)
schema.generate_many_native(count, seed: nil, locale: "en", threads: nil)
schema.generate_to_file(count, path, locale: "en", format: "jsonl", threads: nil)