Skip to content
Kward Search API index

Class: Kward::TiktokenTokenCounter

Inherits:
Object
  • Object
show all
Defined in:
lib/kward/model/context_usage.rb

Overview

Structured context usage result returned to frontends.

Instance Method Summary collapse

Instance Method Details

#count(text, model:) ⇒ Object 



83
84
85
86
87
88
89
90
91
 
# File 'lib/kward/model/context_usage.rb', line 83

def count(text, model:)
  text = text.to_s
  tokenizer = encoding(model)
  return rough_count(text) unless tokenizer.respond_to?(:encode)

  tokenizer.encode(text).length
rescue StandardError
  rough_count(text)
end