Module: Phronomy::Context::TokenEstimator

Defined in:
lib/phronomy/context/token_estimator.rb

Overview

Central, stateless token estimation utility.

All token counting in the framework passes through this module so that the approximation logic lives in one place and can be upgraded without touching any other class.

Default approximation: ceil(char_count / 4). This heuristic is calibrated for ASCII/Latin text (~4 chars/token). For CJK languages (Chinese, Japanese, Korean) the actual token count is approximately 4× higher than the estimate because CJK characters are typically 1 token each in GPT-4/Claude tokenizers (~1 char/token vs the 4 char/token assumed here). Use a tokenizer-backed callable via +.tokenizer=+ for accurate CJK token counting.

Replace the built-in heuristic with any callable via .tokenizer=:

Examples:

Use tiktoken_ruby for accurate GPT token counts

require "tiktoken_ruby"
enc = Tiktoken.encoding_for_model("gpt-4o")
Phronomy::Context::TokenEstimator.tokenizer = ->(text) { enc.encode(text).length }

Reset to built-in heuristic

Phronomy::Context::TokenEstimator.tokenizer = nil

Class Method Summary collapse

Class Method Details

.estimate(input) ⇒ Integer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Estimate the number of tokens for the given input.

Parameters:

  • input (String, Array, #content) —

    a string, a message-like object, or an Array of message-like objects (each must respond to #content).

Returns:

  • (Integer) —

    estimated token count (>= 0)



62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/phronomy/context/token_estimator.rb', line 62

def estimate(input)
  tok = @tokenizer_mutex.synchronize { @tokenizer }
  case input
  when String
    tok ? tok.call(input) : (input.length / 4.0).ceil
  when Array
    input.sum { |m| estimate(m.content.to_s) }
  else
    estimate(input.content.to_s)
  end
end

.reset_tokenizer! ⇒ Object

Resets the tokenizer to the built-in heuristic. Intended for test isolation.



52
53
54
 
# File 'lib/phronomy/context/token_estimator.rb', line 52

def reset_tokenizer!
  @tokenizer_mutex.synchronize { @tokenizer = nil }
end

.tokenizer ⇒ #call?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns:

  • (#call, nil) 


47
48
49
 
# File 'lib/phronomy/context/token_estimator.rb', line 47

def tokenizer
  @tokenizer_mutex.synchronize { @tokenizer }
end

.tokenizer=(callable) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This is a process-wide setting. Set it once at application startup. In tests, call +TokenEstimator.reset_tokenizer!+ after each test to prevent cross-test contamination.

Replace the built-in heuristic with a callable that takes a String and returns an Integer token count. Set to nil to restore the default.

Parameters:

  • callable (#call, nil) 


41
42
43
 
# File 'lib/phronomy/context/token_estimator.rb', line 41

def tokenizer=(callable)
  @tokenizer_mutex.synchronize { @tokenizer = callable }
end