Module: Insika::TokenEstimator

Defined in:
lib/insika/token_estimator.rb

Overview

Cheap token estimate behind an interface. Default: text.length / 4 (off by ~±15%, absorbed by the budget margin). Swappable for a real tokenizer without touching the Builder: any object that responds to #estimate(text) -> Integer works as a replacement (injected at the composition root).

Class Method Summary collapse

Class Method Details

.estimate(text) ⇒ Object 



12
13
14
 
# File 'lib/insika/token_estimator.rb', line 12

def estimate(text)
  text.to_s.length.ceildiv(4)
end