Class: Brute::TokenCounter::Tiktoken

Inherits:
Object
  • Object
show all
Defined in:
lib/brute/token_counter/tiktoken.rb

Overview

Tokens from OpenAI's byte-pair encoder, through the tiktoken_ruby gem.

gem "tiktoken_ruby"
use Brute::Middleware::DefaultCompactionPipeline,
window:        200_000,
token_counter: Brute::TokenCounter::Tiktoken.new

Brute depends on no LLM library, and this is no exception: the gem is required the first time the counter is asked for a number, so an agent that never installs it never pays for it and never hears about it.

It is exact about the text and still approximate about the request -- per_message stands in for the chat-template framing, which is the provider's and not in any encoder.

Constant Summary collapse

ENCODING = 
"o200k_base"

Instance Method Summary collapse

Constructor Details

#initialize(encoding: ENCODING, per_message: 4, encoder: nil) ⇒ Tiktoken

Returns a new instance of Tiktoken.



25
26
27
28
29
 
# File 'lib/brute/token_counter/tiktoken.rb', line 25

def initialize(encoding: ENCODING, per_message: 4, encoder: nil)
  @encoding = encoding
  @per_message = per_message
  @encoder = encoder
end

Instance Method Details

#count(messages, tools: nil) ⇒ Object 



36
37
38
39
40
41
 
# File 'lib/brute/token_counter/tiktoken.rb', line 36

def count(messages, tools: nil)
  warm_up
  text = Rendering.conversation(messages) + Rendering.tools(tools)

  @encoder.encode(text).length + (Array(messages).length * @per_message)
end

#warm_up ⇒ Object

Load the encoder, downloading its vocabulary if it is not cached yet.



32
33
34
 
# File 'lib/brute/token_counter/tiktoken.rb', line 32

def warm_up
  @encoder ||= load_encoder
end