Module: Phronomy::Context::TokenEstimator

Defined in:
lib/phronomy/context/token_estimator.rb

Overview

Central, stateless token estimation utility.

All token counting in the framework passes through this module so that the approximation logic lives in one place and can be upgraded without touching any other class.

Default approximation: ceil(char_count / 4). English text averages ~4 chars/token; Japanese text averages ~2 chars/token so this is a slight underestimate for Japanese.

Replace the built-in heuristic with any callable via .tokenizer=:

Examples:

Use tiktoken_ruby for accurate GPT token counts

require "tiktoken_ruby"
enc = Tiktoken.encoding_for_model("gpt-4o")
Phronomy::Context::TokenEstimator.tokenizer = ->(text) { enc.encode(text).length }

Reset to built-in heuristic

Phronomy::Context::TokenEstimator.tokenizer = nil

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.tokenizer ⇒ Object

Replace the built-in heuristic with a callable that takes a String and returns an Integer token count. Set to nil to restore the default.

Parameters:

  • callable (#call, nil) 


32
33
34
 
# File 'lib/phronomy/context/token_estimator.rb', line 32

def tokenizer
  @tokenizer
end

Class Method Details

.estimate(input) ⇒ Integer

Estimate the number of tokens for the given input.

Parameters:

  • input (String, Array, #content) —

    a string, a message-like object, or an Array of message-like objects (each must respond to #content).

Returns:

  • (Integer) —

    estimated token count (>= 0)



39
40
41
42
43
44
45
46
47
48
 
# File 'lib/phronomy/context/token_estimator.rb', line 39

def estimate(input)
  case input
  when String
    @tokenizer ? @tokenizer.call(input) : (input.length / 4.0).ceil
  when Array
    input.sum { |m| estimate(m.content.to_s) }
  else
    estimate(input.content.to_s)
  end
end