Class: Tiktoken::Encoding

Inherits:
Object
  • Object
show all
Defined in:
lib/tiktoken_ruby/encoding.rb

Constant Summary collapse

CACHE_MUTEX = 
Mutex.new

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#name ⇒ Object (readonly)

Returns the value of attribute name.



6
7
8
 
# File 'lib/tiktoken_ruby/encoding.rb', line 6

def name
  @name
end

Class Method Details

.for_name(encoding) ⇒ Tiktoken::Encoding

This returns a new Tiktoken::Encoding instance for the requested encoding

Parameters:

  • encoding (Symbol) —

    The name of the encoding to load

Returns:



11
12
13
 
# File 'lib/tiktoken_ruby/encoding.rb', line 11

def self.for_name(encoding)
  Tiktoken::Encoding.new(Tiktoken::BpeFactory.send(encoding.to_sym), encoding.to_sym)
end

.for_name_cached(encoding) ⇒ Tiktoken::Encoding

This returns a Tiktoken::Encoding instance for the requested encoding It will reuse an existing encoding if it's already been loaded

Parameters:

  • encoding (Symbol) —

    The name of the encoding to load

Returns:



19
20
21
22
23
24
 
# File 'lib/tiktoken_ruby/encoding.rb', line 19

def self.for_name_cached(encoding)
  CACHE_MUTEX.synchronize do
    @encodings ||= {}
    @encodings[encoding.to_sym] ||= Tiktoken::Encoding.for_name(encoding)
  end
end

Instance Method Details

#decode(tokens, errors: :strict) ⇒ String

Decodes the tokens back into text.

BPE tokens are byte-level, so a single character (an emoji, non-Latin scripts) can span multiple tokens. Truncating a token array to a limit can therefore leave a prefix that is not valid UTF-8. The errors option controls how those invalid byte sequences are handled:

  • :strict (default) - raise Tiktoken::UnicodeError on invalid UTF-8.
  • :replace - substitute each invalid sequence with the Unicode replacement character (U+FFFD, �), matching Python tiktoken's default behavior.

Parameters:

  • tokens (Array<Integer>) —

    The tokens to decode

  • errors (Symbol) (defaults to: :strict) —

    How to handle invalid UTF-8 (:strict or :replace)

Returns:

  • (String) —

    The decoded text (UTF-8)



64
65
66
67
68
69
70
71
72
73
 
# File 'lib/tiktoken_ruby/encoding.rb', line 64

def decode(tokens, errors: :strict)
  case errors
  when :strict
    @ext_base_bpe.decode(tokens)
  when :replace
    decode_bytes(tokens).force_encoding(Encoding::UTF_8).scrub("\u{FFFD}")
  else
    raise ArgumentError, "errors must be :strict or :replace, got #{errors.inspect}"
  end
end

#decode_bytes(tokens) ⇒ String

Decodes the tokens back into their raw bytes, without any UTF-8 validation.

The returned string has ASCII-8BIT (binary) encoding and is not guaranteed to be valid UTF-8 — useful when you want to handle invalid sequences yourself. Matches Python tiktoken's decode_bytes.

Parameters:

  • tokens (Array<Integer>) —

    The tokens to decode

Returns:

  • (String) —

    The decoded bytes (ASCII-8BIT)



83
84
85
 
# File 'lib/tiktoken_ruby/encoding.rb', line 83

def decode_bytes(tokens)
  @ext_base_bpe.decode_bytes(tokens)
end

#encode(text, allowed_special: []) ⇒ Array<Integer>

Encodes the text as a list of integer tokens. This encoding will treat special non text tokens as text unless they're in the allowed_special array. It's basically like the text was escaped

Parameters:

  • text (String) —

    The text to encode

  • allowed_special (Array<String>) (defaults to: []) —

    An array of special tokens to allow

Returns:

  • (Array<Integer>) —

    The encoded tokens



39
40
41
 
# File 'lib/tiktoken_ruby/encoding.rb', line 39

def encode(text, allowed_special: [])
  @ext_base_bpe.encode(text, allowed_special)
end

#encode_ordinary(text) ⇒ Array<Integer>

Encodes the text as a list of integer tokens. This encoding will encode special non text tokens basically it's unescaped

Parameters:

  • text (String) —

    The text to encode

Returns:

  • (Array<Integer>) —

    The encoded tokens



30
31
32
 
# File 'lib/tiktoken_ruby/encoding.rb', line 30

def encode_ordinary(text)
  @ext_base_bpe.encode_ordinary(text)
end

#encode_with_special_tokens(text) ⇒ Array<Integer>

Encodes the text as a list of integer tokens, including special tokens.

Parameters:

  • text (String) —

    The text to encode

Returns:

  • (Array<Integer>) —

    The encoded tokens



46
47
48
 
# File 'lib/tiktoken_ruby/encoding.rb', line 46

def encode_with_special_tokens(text)
  @ext_base_bpe.encode_with_special_tokens(text)
end