Class: Kward::TiktokenTokenCounter

Inherits:

Object

• Object
• Kward::TiktokenTokenCounter

Defined in:

lib/kward/model/context_usage.rb

Overview

Structured context usage result returned to frontends.

Instance Method Summary

• #count(text, model:) ⇒ Object
• #initialize ⇒ TiktokenTokenCounter constructor

  A new instance of TiktokenTokenCounter.

Constructor Details

#initialize ⇒ TiktokenTokenCounter

Returns a new instance of TiktokenTokenCounter.

# File 'lib/kward/model/context_usage.rb', line 83

def initialize
  @encodings = {}
end

Instance Method Details

#count(text, model:) ⇒ Object

# File 'lib/kward/model/context_usage.rb', line 87

def count(text, model:)
  text = text.to_s
  tokenizer = encoding(model)
  return rough_count(text) unless tokenizer.respond_to?(:encode)

  tokenizer.encode(text).length
rescue StandardError
  rough_count(text)
end