Performance Mode

Generate millions of records with parallel processing, streaming, and optimized memory usage.

Quick Start

# Generate 1 million records to file
fake_data_dsl perf User -d schemas/ -c 1000000 -o users.ndjson

# CSV format
fake_data_dsl perf User -d schemas/ -c 1000000 -f csv -o users.csv

# Benchmark mode
fake_data_dsl perf User -d schemas/ --benchmark

CLI Options

Option	Description	Default
-d, --dir DIR	Schema directory	"."
-c, --count N	Number of records	100,000
-o, --output FILE	Output file	stdout
-f, --format FMT	Format: ndjson, json, csv	ndjson
-s, --seed N	Random seed	random
-m, --mode MODE	Generation mode	random
-b, --benchmark	Run benchmark	false

Ruby API

Basic Generation

# Generate records in parallel
records = FakeDataDSL::PerformanceMode.generate(schema, count: 1_000_000)

With Progress Callback

FakeDataDSL::PerformanceMode.generate(schema, count: 1_000_000) do |progress|
  puts "#{progress[:current]} / #{progress[:total]}"
  puts "Rate: #{progress[:rate]} rec/s"
  puts "ETA: #{progress[:eta]}s"
end

Progress hash contains:

• current - Records generated so far
• total - Total records to generate
• percent - Completion percentage
• rate - Records per second
• elapsed - Elapsed time (seconds)
• eta - Estimated time remaining (seconds)

Streaming

Stream records without loading all into memory:

stream = FakeDataDSL::PerformanceMode.stream(schema, count: 10_000_000)

stream.each do |record|
  # Process one record at a time
  process(record)
end

File Output

Write directly to file with streaming:

result = FakeDataDSL::PerformanceMode.to_file(
  schema,
  count: 10_000_000,
  output: "users.ndjson",
  format: :ndjson,
  seed: 42,
  mode: :random
) do |progress|
  puts "#{progress[:percent]}% - #{progress[:rate]} rec/s - ETA: #{progress[:eta]}s"
end

puts "Generated #{result[:records]} in #{result[:elapsed]}s"
puts "Rate: #{result[:rate]} records/second"

Supported formats:

• :ndjson - Newline-delimited JSON (default, best for streaming)
• :json - JSON array
• :csv - CSV with headers

Benchmarking

results = FakeDataDSL::PerformanceMode.benchmark(
  schema,
  counts: [1_000, 10_000, 100_000, 1_000_000]
)

results.each do |count, metrics|
  puts "#{count} records:"
  puts "  Time: #{metrics[:elapsed]}s"
  puts "  Rate: #{metrics[:rate]} rec/s"
  puts "  Memory: #{metrics[:memory_mb]} MB"
end

System Info

Check system capabilities:

info = FakeDataDSL::PerformanceMode.system_info

puts "CPU cores: #{info[:cpu_cores]}"
puts "Optimal threads: #{info[:optimal_threads]}"
puts "Native engine: #{info[:native_available]}"
puts "Recommended batch: #{info[:recommended_batch_size]}"

Configuration

Customize generation behavior:

records = FakeDataDSL::PerformanceMode.generate(
  schema,
  count: 1_000_000,
  batch_size: 10_000,           # Records per batch
  threads: 4,                    # Parallel threads (nil = auto)
  memory_limit_mb: 512,          # Max memory per batch
  use_native: true,              # Use Rust engine if available
  progress_interval: 100_000,    # Report every N records
  gc_interval: 500_000,          # Force GC every N records
  seed: 42,                      # Random seed
  mode: :random                  # Generation mode
)

Performance Tips

1. Use NDJSON for Large Datasets

NDJSON streams records one at a time, minimizing memory:

FakeDataDSL::PerformanceMode.to_file(schema, count: 10_000_000, format: :ndjson, ...)

2. Tune Batch Size

Larger batches = faster, but more memory:

# For machines with lots of RAM
FakeDataDSL::PerformanceMode.generate(schema, count: 1_000_000, batch_size: 50_000)

# For limited memory
FakeDataDSL::PerformanceMode.generate(schema, count: 1_000_000, batch_size: 1_000)

3. Enable Native Engine

The Rust native engine provides 3-4x performance:

# Check if available
FakeDataDSL::PerformanceMode.system_info[:native_available]

# Force native engine
FakeDataDSL::PerformanceMode.generate(schema, count: 1_000_000, use_native: true)

4. Adjust Thread Count

# Use all CPU cores
FakeDataDSL::PerformanceMode.generate(
  schema,
  count: 1_000_000,
  threads: Etc.nprocessors
)

# Leave one core for other work
FakeDataDSL::PerformanceMode.generate(
  schema,
  count: 1_000_000,
  threads: Etc.nprocessors - 1
)

5. Disable Observability for Speed

FakeDataDSL.configure do |config|
  config.metrics_collector = nil
  config.logger = nil
  config.on_field_generated = nil
end

Benchmarks

Typical performance on modern hardware:

Records	NDJSON	JSON	CSV
10K	0.2s	0.3s	0.2s
100K	1.5s	2.0s	1.8s
1M	12s	18s	15s
10M	120s	180s	150s

Results vary based on schema complexity and hardware.

Memory Usage

Approximate memory requirements:

Records	Stream Mode	Batch Mode
100K	~50 MB	~100 MB
1M	~50 MB	~1 GB
10M	~50 MB	~10 GB

Stream mode maintains constant memory regardless of total count.