MultiCompress 🗜️

Gem Version

Status: functional, well-tested, and actively evolving. The current release is suitable for real workloads, but the API and implementation details are still being refined in upcoming releases.

Modern compression technology: zstd, lz4, brotli — unified compression platform with native C performance, fiber-friendly for modern async Ruby stacks.

Bundled library versions in the current release:

  • zstd 1.5.7
  • lz4 1.10.0
  • brotli 1.2.0

📖 Get Started → — Complete technology overview, algorithms, and implementation details

Technology Overview

MultiCompress is a comprehensive compression system that unites three modern algorithms in a single platform. Compared with zlib, results depend on the dataset and algorithm choice: LZ4 is usually chosen for speed, Zstd for balance, and Brotli for ratio.

Algorithm Strength Best for
zstd Best speed/ratio balance Cache, logs, backups
lz4 Fastest compress/decompress IPC, hot cache, real-time
brotli Best ratio for HTTP Web assets, API responses

How It Works

MultiCompress packages modern compression algorithms (zstd, lz4, brotli) with their native C libraries, providing a unified interface. The system includes vendored sources of all compression libraries, eliminating external dependencies.

Key Design Principles

  • Dictionary support: Runtime dictionary use is supported for zstd and brotli; Zstd dictionary training is available in the current release line
  • Zero external dependencies: All C libraries are vendored and compiled
  • Unified API: Same interface for all algorithms — just change the algo: parameter
  • Performance first: Direct bindings to C libraries, minimal overhead
  • Fiber-friendly when available: On Ruby runtimes exposing the Fiber Scheduler C API, compression and decompression cooperate with an active Fiber::Scheduler — safe to use under async, falcon, or similar runtimes without blocking the event loop. On Ruby 2.7.1, the same API works through the normal direct/NOGVL execution paths. See GET_STARTED.md for details and examples.
  • Memory efficient: Streaming support for large datasets, proper resource cleanup
  • Operationally focused: Clear errors, comprehensive tests, and streaming support for practical workloads

Algorithm Auto-Detection

The system can automatically detect compression algorithms when decompressing data:

  • zstd: Detected by magic bytes 28 B5 2F FD (little-endian)
  • lz4: Detected by internal format header validation (custom internal format, NOT compatible with the standard lz4 CLI)
  • brotli: Requires explicit algo: :brotli parameter - no auto-detection

Important: Auto-detection only works for ZSTD and LZ4. Brotli data must be decompressed with explicit algorithm specification.

Security: Decompression now enforces a default 512MB one-shot output cap, a default 2GB cumulative streaming cap, and a 32MB dictionary file size cap.

Security limits

Decompression-facing APIs now use separate size defaults for one-shot and streaming paths:

  • One-shot default output cap: 512MB
  • Streaming cumulative cap: 2GB across the lifetime of an Inflater/Reader
  • Global configuration: MultiCompress.configure
  • Per-call override: max_output_size:
  • Dictionary file size cap: 32MB for MultiCompress::Dictionary.load

Examples:

MultiCompress.configure do |config|
  config.max_output_size = 512 * 1024 * 1024
  config.streaming_max_output_size = 2 * 1024 * 1024 * 1024
end

MultiCompress.decompress(blob, algo: :zstd, max_output_size: 64 * 1024 * 1024)

MultiCompress::Reader.open("archive.zst", max_output_size: 128 * 1024 * 1024) do |reader|
  puts reader.read
end

If max_output_size: is omitted, one-shot calls use MultiCompress.config.max_output_size and streaming calls use MultiCompress.config.streaming_max_output_size.

Algorithm Comparison

Algorithm Speed Ratio Best Use Case
lz4 Fastest Good Real-time processing, IPC, hot cache paths
zstd Fast Excellent General purpose, logs, backups, web APIs
brotli Slower Best Static assets, CDN, long-term storage

Benchmark Results

Benchmark numbers are environment-dependent. The tables below were generated with MultiCompress 0.3.2 using vendored zstd 1.5.7, lz4 1.10.0, and brotli 1.2.0 on Ruby 3.1.7 / arm64-darwin24, 200 iterations per test.

Use these numbers as relative guidance, not absolute guarantees. In this run, lz4 is consistently the fastest, zstd gives the best speed/ratio balance, and brotli gives the smallest output size but is not optimized for speed-first paths.

Performance comparison against Ruby's built-in zlib compression:

🗜️ Compression ratio (%, lower is better)

Configuration zlib lz4 zstd brotli
Small JSON (~10KB, GC) 9.4% 16.1% 6.9% 6.1%
Small text (~10KB, GC) 3.1% 4.6% 3.2% 2.6%
Small JSON (~10KB, no GC) 9.4% 16.1% 6.9% 6.1%
Small text (~10KB, no GC) 3.1% 4.6% 3.2% 2.6%
Medium JSON (~370KB, GC) 8.5% 15.7% 6.6% 5.4%
Medium logs (~168KB, GC) 8.6% 17.2% 5.3% 3.3%
Medium JSON (~370KB, no GC) 8.5% 15.7% 6.6% 5.4%
Medium logs (~168KB, no GC) 8.6% 17.2% 5.3% 3.3%
Large JSON (~1.6MB, GC) 8.1% 15.1% 6.1% 5.6%
Large logs (~600KB, GC) 7.6% 16.0% 2.9% 2.0%
Large JSON (~1.6MB, no GC) 8.1% 15.1% 6.1% 5.6%
Large logs (~600KB, no GC) 7.6% 16.0% 2.9% 2.0%

⚡ Total time (compress + decompress, ms — lower is faster)

Configuration zlib lz4 zstd brotli
Small JSON (~10KB, GC) 0.05 0.01 0.02 0.14
Small text (~10KB, GC) 0.04 0.00 0.01 0.12
Small JSON (~10KB, no GC) 0.05 0.01 0.02 0.14
Small text (~10KB, no GC) 0.04 0.00 0.01 0.11
Medium JSON (~370KB, GC) 2.60 0.29 0.41 2.45
Medium logs (~168KB, GC) 1.28 0.13 0.17 0.96
Medium JSON (~370KB, no GC) 2.62 0.27 0.39 2.41
Medium logs (~168KB, no GC) 1.19 0.13 0.17 1.08
Large JSON (~1.6MB, GC) 11.60 1.30 1.81 11.12
Large logs (~600KB, GC) 4.11 0.41 0.46 2.99
Large JSON (~1.6MB, no GC) 11.26 1.24 1.77 10.77
Large logs (~600KB, no GC) 4.01 0.42 0.45 2.90

📊 Speedup vs zlib by total time (higher is better)

Configuration zlib lz4 zstd brotli
Small JSON (~10KB, GC) 1.00x 5.00x 2.50x 0.36x
Small text (~10KB, GC) 1.00x N/A 4.00x 0.33x
Small JSON (~10KB, no GC) 1.00x 5.00x 2.50x 0.36x
Small text (~10KB, no GC) 1.00x N/A 4.00x 0.36x
Medium JSON (~370KB, GC) 1.00x 8.97x 6.34x 1.06x
Medium logs (~168KB, GC) 1.00x 9.85x 7.53x 1.33x
Medium JSON (~370KB, no GC) 1.00x 9.70x 6.72x 1.09x
Medium logs (~168KB, no GC) 1.00x 9.15x 7.00x 1.10x
Large JSON (~1.6MB, GC) 1.00x 8.92x 6.41x 1.04x
Large logs (~600KB, GC) 1.00x 10.02x 8.93x 1.37x
Large JSON (~1.6MB, no GC) 1.00x 9.08x 6.36x 1.05x
Large logs (~600KB, no GC) 1.00x 9.55x 8.91x 1.38x

📏 Compressed size (bytes, lower is better)

Configuration zlib lz4 zstd brotli
Small JSON (~10KB, GC) 900 1544 665 583
Small text (~10KB, GC) 310 461 322 256
Small JSON (~10KB, no GC) 900 1544 665 583
Small text (~10KB, no GC) 310 461 322 256
Medium JSON (~370KB, GC) 31524 57986 24557 20122
Medium logs (~168KB, GC) 14488 28950 8985 5549
Medium JSON (~370KB, no GC) 31524 57986 24557 20122
Medium logs (~168KB, no GC) 14488 28950 8985 5549
Large JSON (~1.6MB, GC) 133275 250026 100965 92591
Large logs (~600KB, GC) 45432 96130 17385 12250
Large JSON (~1.6MB, no GC) 133275 250026 100965 92591
Large logs (~600KB, no GC) 45432 96130 17385 12250

Dependencies for benchmarking:

  • memory_profiler — Memory usage analysis
  • benchmark-ips — Iterations per second benchmarking

Or use the build script:

./build.sh

Command-line tool & DB column compression

Installing the gem provides a gzip-style multi_compress executable:

multi_compress file.json          # -> file.json.zst
multi_compress -d file.json.zst   # -> file.json
cat big.log | multi_compress -a zstd -c > big.log.zst

MultiCompress::Codec is a general-purpose application-side envelope for a single database column (optional ActiveRecord Type/Coder adapters and optional Base64 for text columns). It is not the format for SQL-side inspection.

For MySQL 5.7 or PostgreSQL + DBeaver, use the SQL-readable database API. MCDB1 is dictionary-free; MCDB2 is opt-in for a homogeneous column with an immutable zstd dictionary registry:

require "multi_compress/database"

blob = MultiCompress::Database.compress("JSON or UTF-8 text") # store in LONGBLOB / bytea
text = MultiCompress::Database.decompress(blob)

For repeated small JSON/text records with the same shape, use MCDB2 and keep the dictionary as append-only application data:

dictionary = MultiCompress::Database::Dictionary.train(samples, id: 42)
blob = MultiCompress::Database.compress(payload_json, dictionary: dictionary)

See docs/database-envelope-v2.md for the registry, generated view, rollout and DBeaver flow.

For database-side reading, the installed gem creates a self-contained DBA bundle from the exact gem version — no Git clone, manual .so copy, or hand-written extension SQL:

bundle exec multi_compress db package postgres --output tmp/multi-compress-postgres.tar.gz
# or
bundle exec multi_compress db package mysql --output tmp/multi-compress-mysql.tar.gz

The DBA extracts that archive on the database host and runs make verify, make doctor, and sudo make install. PostgreSQL enablement also requires the migration and read roles; MySQL uses a local Unix socket and has explicit upgrade, disable, and uninstall confirmations. The bundle compiles the native reader against that host's real PostgreSQL/MySQL environment and installs it into the server-reported directory. Generate the DBeaver-facing read view from the application repository as well:

bundle exec multi_compress db view postgres \
  --table app.events --column payload_compressed \
  --view admin.events_readable --columns id,created_at,status \
  --output db/views/events_readable.sql

Use --extension-schema NAME with the PostgreSQL view generator only when the DBA enabled the extension outside the default multi_compress schema. See GET_STARTED.md for the full app-server → DBA → DBeaver flow.

The CLI writes this gem's internal LZ4 format as .mclz4 (not .lz4), since it is not interchangeable with the standard lz4 CLI.

Requirements

  • Ruby >= 2.7.1
  • C compiler (gcc, clang)

License

MIT — see LICENSE.txt.

For MCDB2 registry DDL, pass --payload-table, --payload-column, and --payload-dictionary-id-column to multi_compress db registry; this creates the payload FK and enforces header dictionary-reference consistency.