Module: Pikuri::VectorDb::Tokenizer

Defined in:

lib/pikuri/vector_db/tokenizer.rb,
lib/pikuri/vector_db/tokenizer/llama_server.rb,
lib/pikuri/vector_db/tokenizer/char_heuristic.rb

Overview

Namespace for tokenizers. Two ship: CharHeuristic (zero-dep default, ~4 chars/token — slight overshoot is fine, embedders truncate gracefully) and LlamaServer (exact via /tokenize, typically the embedder's own server so chunk sizing matches its vocab).

Tokenizer protocol

Duck-typed, single method (like pikuri's other seams): +#count(text) -> Integer+, 0 for empty, deterministic. Lifting it out of Chunker lets the chunker test against a fake, lets a future Tiktoken plug in untouched, and makes the CharHeuristic/LlamaServer choice a host knob (the chunker: passed to Extension).

Defined Under Namespace

Classes: CharHeuristic, LlamaServer