static_embeddings

A Ruby runtime for converted Model2Vec / potion static embedding models. A model is converted once into a local .semb file and loaded through a small C extension. No ONNX Runtime, no Rust or Python at runtime, no network access, one mmap-able file, binary float32 output.

The current production target is minishlab/potion-retrieval-32M.

require "static_embeddings"

model = StaticEmbeddings.load_model("potion-retrieval-32m")

blob = model.embed("postgres pipeline mode in Ruby")
blob.bytesize == model.dim * 4

batch = model.embed_batch([
  "postgres pipeline mode in Ruby",
  "local static embeddings without ONNX Runtime"
])

New to the gem? Start with GET_STARTED.md. Design rationale is in docs/ARCHITECTURE.md.

Runtime contract

The runtime loads only .semb files produced by this repository's converter, never arbitrary HuggingFace models. A .semb file carries validated metadata, a BERT WordPiece tokenizer profile, an mmap-ready vocabulary lookup, float32 embedding rows, and provenance plus a checksum.

The only supported tokenizer profile is BERT_WORDPIECE_V1. The converter rejects unsupported tokenizer features rather than approximating them.

Converting a model

git lfs install
git clone https://huggingface.co/minishlab/potion-retrieval-32M

bundle exec rake compile
bundle exec ruby -Ilib exe/static_embeddings convert ./potion-retrieval-32M \
  --id potion-retrieval-32m

The result lands in ~/.cache/static_embeddings/models/potion-retrieval-32m.semb. Inspect or verify it with the inspect and verify subcommands, then load it:

model = StaticEmbeddings.load_model("potion-retrieval-32m")
model = StaticEmbeddings.load(ENV.fetch("EMBEDDING_MODEL"))   # explicit path

Convert during image build or deploy preparation; production should only ever see .semb files.

Verify once, not on every boot

load does not check the SHA-256 by default: hashing a 135 MB file at boot would undo the point of mmapping it lazily. verify streams the file, so it costs one chunk of memory rather than a copy of the model. Structural validation always runs, so a malformed container is rejected, but a bit flip inside the matrix is not detected and silently changes vectors. Verify where the artifact enters your system:

StaticEmbeddings.load(path, verify: true)   # image build / CI
StaticEmbeddings.load(path)                 # hot path, artifact already trusted

API

model.path
model.dim
model.vocab_size
model.max_tokens
model.normalized?
model.lowercase?
model.unk_id
model.mapped_bytes
model.provenance
model.model_id

vector_blob = model.embed("postgres pipeline mode in Ruby")               # f32 by default
small_blob  = model.embed("postgres pipeline mode in Ruby", format: :f16) # half-size storage
bounded     = model.embed(huge_text, validate_encoding: :prefix)          # see "Large inputs"
array       = model.embed_array("postgres pipeline mode in Ruby")

batch_blob  = model.embed_batch(texts)
small_batch = model.embed_batch(texts, format: :f16)
arrays      = model.embed_batch_arrays(texts)

stats = model.embed_with_stats("postgres pipeline mode in Ruby")
stats[:vector]
stats[:token_count]
stats[:unk_count]
stats[:truncated]

ids         = model.tokenize("postgres pipeline mode in Ruby")
vector_blob = model.embed_token_ids(ids)
stats       = model.embed_token_ids_with_stats(ids)

model.cosine_top_k(query_blob, matrix_blob, 10)
model.dot_top_k(query_blob, matrix_blob, 10)

StaticEmbeddings.cosine_top_k(query_blob, matrix_blob, 10, dim: model.dim)
StaticEmbeddings.pack(rows, format: :f16)     # Array(s) of Float -> blob
StaticEmbeddings.unpack(blob, model.dim)      # blob -> Array of Array(Float)

Output is a binary String of little-endian float32 values in row-major order. embed_array and embed_batch_arrays decode that into Ruby Float objects and exist for debugging and application code, not for the hot path.

embed_token_ids pools ids you supply, applying the same max_tokens truncation as embed, so embed_token_ids(model.tokenize(text)) equals embed(text). Pass max_tokens: false to pool every id. It is for reusing a cached tokenization and for benchmarking pooling in isolation; it is not a faster path for ordinary text.

format: :f16

format: selects the returned storage encoding only. The model always computes in float32. For a 512-dimensional model one vector goes from 2048 to 1024 bytes. It is accepted by embed, embed_batch, embed_with_stats, embed_token_ids, embed_token_ids_with_stats, pack, unpack, and the top-k helpers.

Choose f16 for the RAM and storage it saves, not for speed. It halves the bytes a top-k scan streams, but every row still has to be decoded before scoring, and which effect wins is a property of the machine — this repository's own samples show f16 winning on x86-64 F16C and losing on M1 Pro. See docs/PERFORMANCE.md before assuming either.

StaticEmbeddings.simd_backend reports the live kernel: "neon-fp16", "f16c", or "lut" for the lookup-table fallback, which is several-fold slower. Check it before drawing any conclusion from an f16 benchmark.

Similarity helpers

cosine_top_k divides by both norms and returns similarity in [-1, 1]. It raises ArgumentError on a zero-norm query and scores zero-norm rows as 0.0. dot_top_k returns the raw dot product; use it when your vectors are already unit length, including output from a model whose normalized? is true.

Both skip rows scoring NaN rather than admitting them to the result, so a corrupt row is never reported as if it were merely empty.

Three contracts, each of which raises rather than guessing:

• dim: is required on the module-level form. A blob is bare bytes with no dimension and no format tag, so an f32 query against an f16 matrix would otherwise divide out to a plausible row count and return nonsense. model.cosine_top_k and model.dot_top_k fill dim: in and are the recommended form.
• A matrix at or above 1 MiB must be frozen. It is scanned with the GVL released so other threads keep running, and a frozen String cannot be mutated mid-scan. allow_unfrozen: true scans while holding the GVL: correct, but it blocks every other thread for the duration.
• A large format: :f32 matrix must be 4-byte aligned. Anything from pack, embed_batch or File.binread is; a byteslice at an odd offset may not be.

MATRIX = model.embed_batch(corpus).freeze
model.dot_top_k(query, MATRIX, 10)          # lock-free, concurrent, GVL released

Large inputs

When truncation is active the runtime does not copy the whole string into C memory. It copies a leading slice sized from max_tokens, cut on a word boundary, and grows the budget only if that slice did not reach max_tokens. The result is identical to tokenizing the whole document, because WordPiece segmentation is word-local, but the bytes the tokenizer reads stop growing with input size: with max_tokens: 512 the window is a few kilobytes whether the document is 10 KB or 3 MB.

That bounds the tokenizer, not the whole call. embed also has to establish that the Ruby String is valid UTF-8, and when Ruby has not computed the string's coderange yet that scan is O(total bytes) and happens before any prefix is chosen:

text = File.read(path)   # coderange unknown
model.embed(text)        # full UTF-8 scan, then a few KB of tokenizing
model.embed(text)        # coderange cached: prefix window only

Three ways out, in order of preference. Reuse the String, since Ruby caches the coderange after the first scan. Force the scan once outside the hot path with text.valid_encoding?. Or ask for validation to be bounded the same way tokenizing is:

model.embed(text, validate_encoding: :prefix)

:prefix skips the up-front scan and lets the tokenizer validate the bytes it actually reads; malformed UTF-8 inside the window still raises. On a 3 MB document that turned 442 µs into 78 µs, against 71 µs for the same string with its coderange already cached.

The trade is real and worth stating: under :prefix, invalid bytes past the truncation window are never looked at and no longer raise. :full is the default and keeps the whole-string guarantee. A coderange Ruby has already computed is honoured either way, so a String already known to be broken raises in both modes, and one already known to be valid costs nothing in either. The option is accepted by embed, embed_batch, embed_with_stats and tokenize.

Text with no legal cut anywhere in the scan window — one enormous word, a run of combining marks — falls back to copying the whole input. With max_tokens: false there is nothing to truncate, so the whole input is copied. docs/ARCHITECTURE.md explains what makes a cut legal.

embed_batch builds its result in C memory and then copies it into a Ruby String, so peak memory for one call is about twice the returned blob. Chunk very large corpora.

Concurrency

embed_batch snapshots its input, then releases the GVL for the C work, so one request can compute embeddings while other Puma threads keep serving. Because the snapshot is taken up front, mutating the Array afterwards does not corrupt anything but is ignored.

cosine_top_k and dot_top_k scan a frozen matrix without any lock, so any number of threads can search one shared corpus at once.

The runtime does not expose internal parallelism through threads:. For offline indexing, split work at the job level and run multiple Ruby workers.

With a Fiber::Scheduler installed and the current fiber non-blocking, work above 2 KB is handed to a real thread the fiber joins, so the scheduler keeps running other fibers. That is one OS thread per call: under high concurrency, prefer batching over many small embed calls.

For Puma with preload_app!, load and warm before workers fork:

preload_app!

before_fork do
  MODEL = StaticEmbeddings.load(ENV.fetch("EMBEDDING_MODEL"))
  MODEL.warmup!
end

Language safety

potion-retrieval-32M is an English retrieval model. Russian text can produce a high [UNK] ratio while still returning a valid vector, which is the most likely way to end up with a silently bad index. Check the ratio before trusting a corpus:

stats = model.embed_with_stats(text)
ratio = stats[:token_count].zero? ? 0.0 : stats[:unk_count].to_f / stats[:token_count]

warn "high [UNK] ratio: #{ratio.round(3)}" if ratio > 0.3

Out-of-vocabulary text is also slower, because every word falls through to subword splitting instead of hitting the vocabulary directly.

Correctness contract

The reference implementation is model2vec.StaticModel. The converter records its decisions in the .semb file: truncation at 512 tokens by default, applied after tokenization and before pooling; [UNK] tokens dropped; a zero vector for input with no usable tokens; L2 normalization when the source model requires it.

Token ids must match the reference exactly. Vectors are compared with tolerance, because floating-point addition order is not bit-stable across implementations:

cosine >= 1 - 1e-6
max_abs_diff < 1e-5

docs/MODEL_AUDIT.md records the parity result, digests and edge-case behaviour for every trusted conversion, and the command to reproduce it. A new conversion is not trusted until it has its own record.

Development

bundle install
bundle exec rake            # compile + fixtures + test
bundle exec rake demo_model # tiny synthetic model for smoke tests

StaticEmbeddings.load_builtin loads that demo model. It only works inside a checkout, it is not shipped in the published gem, and StaticEmbeddings.builtin_available? returns false when it is missing. It is a test fixture and API demo, not a retrieval quality baseline.

ruby tools/benchmark.rb                  # normalised performance budget
bundle exec rake cancellation_timing     # timing-sensitive, excluded from rake test
samples/run_all.sh                       # native hot-path probes; see samples/README.md

The C memory smoke binary runs without Ruby:

ruby tools/make_fixture_model.rb
ruby -Ilib -e 'require "static_embeddings"; StaticEmbeddings.convert("test/fixtures/tiny-wordpiece", output_path: "tmp/test-tiny.semb", model_id: "fixture/tiny-wordpiece")'
cc -O2 -std=c99 -Wall -Wextra -Iext/static_embeddings \
  tools/memory_smoke.c \
  ext/static_embeddings/se_f16.c \
  ext/static_embeddings/se_topk.c \
  ext/static_embeddings/se_format.c \
  ext/static_embeddings/se_unicode.c \
  ext/static_embeddings/se_tokenizer.c \
  ext/static_embeddings/se_embed.c \
  -lm -o tmp/memory_smoke
./tmp/memory_smoke tmp/test-tiny.semb

The runtime is intentionally narrow. If a model needs unsupported tokenizer behaviour, fix the converter whitelist and audit it; do not make the C runtime guess.

License

MIT. See LICENSE.txt.