NNQ::Zstd — transparent Zstd compression for NNQ sockets

CI Gem Version License: ISC Ruby

Wraps any NNQ::Socket with transparent per-message Zstd compression, sender-side dictionary training, and in-band dictionary shipping. No handshake, no negotiation — both peers must wrap for it to work.

See RFC.md for the normative wire protocol.

Quick start

require "nnq"
require "nnq/zstd"

push = NNQ::PUSH0.new
push.connect("tcp://127.0.0.1:5555")
push = NNQ::Zstd.wrap(push, level: -3)   # fast (-3) or balanced (3)

push.send("payload")                      # compressed on the wire

pull = NNQ::PULL0.new
pull.bind("tcp://*:5555")
pull = NNQ::Zstd.wrap(pull)               # receiver-only: no dict config

pull.receive                              # => "payload"

How it works

  • Every message carries a 4-byte preamble:
    • 00 00 00 00 — uncompressed plaintext (preamble stripped on recv).
    • Zstd frame magic — a full Zstd frame (compressed).
    • Zstd dict magic — a Zstd dictionary to install (not surfaced to the app).
  • The sender trains a dict from its first ~1000 small messages (or 100 KiB cumulative sample bytes), then compresses subsequent small messages with it. The dict is shipped in-band before the next real payload.
  • User-supplied dictionaries may be passed via dict:: ruby NNQ::Zstd.wrap(socket, level: -3, dict: [dict_bytes_1, dict_bytes_2]) All supplied dicts are shipped to peers on the wire. Training is skipped when dict: is given.
  • Decompression is bounded by min(16 MiB, recv_maxsz). Frames whose header omits Frame_Content_Size are rejected.
  • Up to 32 dicts and 128 KiB cumulative per wrapper. Any protocol violation raises NNQ::Zstd::ProtocolError.

Out of scope

  • Negotiation / auto-detection. Both peers must wrap.
  • Dict persistence across process restarts. Training is per-session.
  • Receiver-side training — receivers only install shipped dicts.

License

ISC. See LICENSE.