Class: Brute::TokenCounter::Approximate

Inherits:
Object
  • Object
show all
Defined in:
lib/brute/token_counter/approximate.rb

Overview

Tokens from text length, at a flat ratio of characters to tokens.

Four characters to the token is the usual approximation. It needs no dependency and it is close enough to decide what to give up; put a Tiktoken counter on env when the difference matters.

per_message is what the rendering cannot see: every message is wrapped in the provider's own chat template on the way out, and that framing costs a few tokens each whatever the message says.

Instance Method Summary collapse

Constructor Details

#initialize(chars_per_token: 4.0, per_message: 4) ⇒ Approximate

Returns a new instance of Approximate.



18
19
20
21
22
23
24
25
 
# File 'lib/brute/token_counter/approximate.rb', line 18

def initialize(chars_per_token: 4.0, per_message: 4)
  unless chars_per_token.positive?
    raise ArgumentError, "chars_per_token must be greater than 0, got #{chars_per_token}"
  end

  @chars_per_token = chars_per_token
  @per_message = per_message
end

Instance Method Details

#count(messages, tools: nil) ⇒ Object 



27
28
29
30
31
 
# File 'lib/brute/token_counter/approximate.rb', line 27

def count(messages, tools: nil)
  text = Rendering.conversation(messages) + Rendering.tools(tools)

  (text.length / @chars_per_token).to_i + (Array(messages).length * @per_message)
end