Module: Ask::TokenUsage::Counting

Defined in:
lib/ask/token_usage/counting.rb

Overview

Real token measurement via tiktoken_ruby (OpenAI's BPE tokenizer). This is what turns "a page of text" into a countable, sellable number.

Constant Summary collapse

DEFAULT_ENCODING = 
"cl100k_base"

Class Method Summary collapse

Class Method Details

.count(text, model: nil) ⇒ Object

Count the tokens in text. Pass model: for a model-aware encoding. Unknown models fall back to the default encoding; if even tiktoken fails we estimate at ~4 chars per token rather than blowing up.



18
19
20
21
22
 
# File 'lib/ask/token_usage/counting.rb', line 18

def count(text, model: nil)
  encoding_for(model).encode(text.to_s).length
rescue StandardError
  estimate(text.to_s)
end

.encoding_for(model) ⇒ Object 



24
25
26
27
28
29
30
31
 
# File 'lib/ask/token_usage/counting.rb', line 24

def encoding_for(model)
  if model
    enc = Tiktoken.encoding_for_model(model)
    return enc if enc
  end

  Tiktoken.get_encoding(DEFAULT_ENCODING)
end

.estimate(text) ⇒ Object

Coarse fallback estimate (~4 characters per token).



34
35
36
 
# File 'lib/ask/token_usage/counting.rb', line 34

def estimate(text)
  (text.to_s.length / 4.0).ceil
end