Module: Brute::TokenCounter

Defined in:
lib/brute/token_counter.rb,
lib/brute/token_counter/tiktoken.rb,
lib/brute/token_counter/approximate.rb

Overview

How big a conversation is, in tokens.

A counter is anything answering count(messages, tools: nil). The tools are part of the question because their schemas ride in every request alongside the messages -- an agent carrying a dozen of them is spending context on JSON Schema before anyone has said a word.

counter = Brute::TokenCounter::Approximate.new
counter.count(env[:messages], tools: env[:tools])

What a turn is currently costing is a different question, and estimate answers it: the provider counted the conversation exactly when it answered, so that number is trusted and only what has been appended since is counted locally.

Brute::TokenCounter.estimate(env)

Defined Under Namespace

Modules: Rendering Classes: Approximate, Tiktoken

Class Method Summary collapse

Class Method Details

.default ⇒ Object 



26
 
# File 'lib/brute/token_counter.rb', line 26

def self.default = Approximate.new

.estimate(env, counter: nil, tools: nil) ⇒ Object

What the whole turn costs right now.

:counter defaults to the one the turn already decided on (env), :tools to the ones the pipeline advertises (env).

The reported total already covers the system prompt, the tool schemas and the provider's own chat-template overhead, so the warm path adds only the messages that landed after the reply it describes -- and never the schemas, which are already inside it. Anything else double counts.



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
 
# File 'lib/brute/token_counter.rb', line 38

def self.estimate(env, counter: nil, tools: nil)
  counter ||= env[:token_counter] || default
  tools = env[:tools] if tools.nil?
  messages = env[:messages] || []
  reported = env.dig(:metadata, :last_llm_usage)&.total.to_i
  answered = messages.rindex { |message| message.role == :assistant }

  # Nothing sent yet, or a conversation the reported number no longer
  # describes -- a compaction that left no reply behind, say. Count it all.
  if reported.zero? || answered.nil?
    counter.count(messages, tools: tools)
  else
    reported + counter.count(messages[(answered + 1)..])
  end
end