Class: Pikuri::VectorDb::Tokenizer::CharHeuristic

Inherits:
Object
  • Object
show all
Defined in:
lib/pikuri/vector_db/tokenizer/char_heuristic.rb

Overview

Approximate tokenization via a fixed chars-per-token ratio — the zero-dep default, no network round-trip. Overshoot is fine because embedders truncate gracefully (a 10%-over chunk is capped, not rejected).

English BPE tokenizers (GPT, llama, BGE, sentence-transformers) settle around 3.5–4.5 chars/token on prose; code drops to ~3, CJK to ~1–2 (one logograph per token). Default 4 is the English sweet spot — override chars_per_token: for other corpora, or use LlamaServer for exact counts when the corpus is CJK/code-heavy or the embedder's window is tight enough that overshoot clips content.

Instance Method Summary collapse

Constructor Details

#initialize(chars_per_token: 4.0) ⇒ CharHeuristic

Parameters:

  • chars_per_token (Numeric) (defaults to: 4.0) —

    characters per token the approximation assumes. Default 4.0 — English prose against a BPE tokenizer. Pass a smaller number for denser scripts (Mandarin ≈ 2; Japanese ≈ 1.5).

Raises:

  • (ArgumentError) 


22
23
24
25
26
 
# File 'lib/pikuri/vector_db/tokenizer/char_heuristic.rb', line 22

def initialize(chars_per_token: 4.0)
  raise ArgumentError, 'chars_per_token must be positive' if chars_per_token <= 0

  @chars_per_token = chars_per_token.to_f
end

Instance Method Details

#count(text) ⇒ Integer

Approximate token count. Empty string returns 0; short strings round up to at least 1 token.

Parameters:

  • text (String)

Returns:

  • (Integer) —

    approximate token count, >= 0.



33
34
35
36
37
 
# File 'lib/pikuri/vector_db/tokenizer/char_heuristic.rb', line 33

def count(text)
  return 0 if text.empty?

  (text.length / @chars_per_token).ceil
end