Module: Gigatoken::Encodings

Defined in:
lib/gigatoken/encodings.rb

Overview

The tiktoken encodings gigatoken vendors ranks for, and the pieces a .tiktoken file doesn't carry: its pretokenizer scheme and special-token table (see Tokenizer.from_tiktoken and lib/gigatoken/encodings/ PROVENANCE.md, the source of truth this is transcribed from).

Constant Summary collapse

NAMES =

The packaged encoding names — the single source error messages naming what's available are built from (the same discipline Native.pretokenizer_names / PretokenizerType::NAMES applies to pretokenizer scheme names).

REGISTRY.keys.freeze

Class Method Summary collapse

Class Method Details

.[](name) ⇒ Object

The pretokenizer:, special_tokens: registered for a packaged encoding name, or nil.



92
93
94
 
# File 'lib/gigatoken/encodings.rb', line 92

def [](name)
  REGISTRY[name]
end

.unpackable_reason(name) ⇒ Object

Why name can't be packaged, or nil when there's no reason on record (it's either packaged, or simply not one gigatoken knows of).



98
99
100
 
# File 'lib/gigatoken/encodings.rb', line 98

def unpackable_reason(name)
  UNPACKABLE_REASONS[name]
end