Module: RubynCode::Observability::TokenCounter

Defined in:
lib/rubyn_code/observability/token_counter.rb

Overview

Estimates token counts from text using a character-based heuristic.

This provides a fast approximation (~4 characters per token) suitable for budget tracking and context-window management. For exact counts, use the API’s reported usage fields instead.

Constant Summary collapse

CHARS_PER_TOKEN =

Average characters per token for English text and source code.

4

Class Method Summary collapse

Class Method Details

.estimate(text) ⇒ Integer

Estimates the token count for a given string.

Parameters:

  • text (String, nil) —

    the text to estimate

Returns:

  • (Integer) —

    estimated token count (minimum 0)



21
22
23
24
25
 
# File 'lib/rubyn_code/observability/token_counter.rb', line 21

def estimate(text)
  return 0 if text.nil? || text.empty?

  (text.bytesize.to_f / CHARS_PER_TOKEN).ceil
end

.estimate_messages(messages) ⇒ Integer

Estimates the token count for an array of messages by serializing them to JSON first. Accounts for the structural overhead of message formatting (role tags, separators, etc.).

Parameters:

  • messages (Array<Hash>) —

    messages in the API conversation format

Returns:

  • (Integer) —

    estimated token count (minimum 0)



33
34
35
36
37
38
 
# File 'lib/rubyn_code/observability/token_counter.rb', line 33

def estimate_messages(messages)
  return 0 if messages.nil? || messages.empty?

  json = JSON.generate(messages)
  estimate(json)
end