Class: Gigatoken::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/gigatoken/tokenizer.rb

Overview

A tokenizer: encode, batch encode, decode, and vocabulary introspection over a native Gigatoken::Native::BPETokenizer or Gigatoken::Native::SentencePieceTokenizer.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(native, special_tokens: {}) ⇒ Tokenizer

Returns a new instance of Tokenizer.



92
93
94
95
 
# File 'lib/gigatoken/tokenizer.rb', line 92

def initialize(native, special_tokens: {})
  @native = native
  @special_tokens = special_tokens
end

Instance Attribute Details

#special_tokens ⇒ Object (readonly)

Returns the value of attribute special_tokens.



152
153
154
 
# File 'lib/gigatoken/tokenizer.rb', line 152

def special_tokens
  @special_tokens
end

Class Method Details

.from_encoding(name) ⇒ Object

Load one of the tiktoken encodings gigatoken vendors ranks for, by name — see Gigatoken::Encodings::NAMES — entirely from the vendored files: no network, no writable cache.

Raises:



41
42
43
44
45
46
47
48
 
# File 'lib/gigatoken/tokenizer.rb', line 41

def self.from_encoding(name)
  encoding = Encodings[name]
  return from_tiktoken(encoding[:rank_file], pretokenizer: encoding[:pretokenizer], special_tokens: encoding[:special_tokens]) if encoding

  reason = Encodings.unpackable_reason(name)
  detail = reason ? " — #{reason}" : ""
  raise Error, "#{name.inspect}: not a packaged encoding#{detail} (packaged encodings: #{Encodings::NAMES.join(", ")})"
end

.from_file(path) ⇒ Object

Load from a tokenizer.json path, or a directory containing one.



22
23
24
25
 
# File 'lib/gigatoken/tokenizer.rb', line 22

def self.from_file(path)
  path = File.join(path, "tokenizer.json") if File.directory?(path)
  from_json(File.binread(path))
end

.from_hub(repo_id, revision: "main", hub: Hub.new) ⇒ Object

Load tokenizer.json from HuggingFace Hub repo repo_id at revision (downloaded directly; huggingface_hub is not required).



52
53
54
 
# File 'lib/gigatoken/tokenizer.rb', line 52

def self.from_hub(repo_id, revision: "main", hub: Hub.new)
  from_file(hub.hub_file(repo_id, "tokenizer.json", revision: revision))
end

.from_json(data) ⇒ Object

Load from in-memory tokenizer.json contents (String or bytes). Backed by a BPETokenizer or a SentencePieceTokenizer, per the model's byte_fallback flag.



16
17
18
19
 
# File 'lib/gigatoken/tokenizer.rb', line 16

def self.from_json(data)
  native = Native.load_hf_json(data)
  new(native, special_tokens: special_tokens_from_json(data))
end

.from_tiktoken(path, pretokenizer:, special_tokens: {}) ⇒ Object

Load from a .tiktoken mergeable-ranks file. The file carries neither a pretokenization scheme nor special tokens — nothing is guessed here, so pretokenizer: is required: one of the schemes gigatoken ships (see Native.pretokenizer_names, e.g. "gpt2"/"r50k", "gpt4"/"cl100k", "o200k", "qwen2", "qwen35", "olmo3", "deepseek_v3", "nemotron", "kimi"). special_tokens: maps token content to id (none by default).



33
34
35
36
 
# File 'lib/gigatoken/tokenizer.rb', line 33

def self.from_tiktoken(path, pretokenizer:, special_tokens: {})
  native = Native::BPETokenizer.from_tiktoken(path.to_s, pretokenizer, special_tokens)
  new(native, special_tokens: special_tokens)
end

.load(source, pretokenizer: nil, special_tokens: {}, revision: "main", hub: Hub.new) ⇒ Object

Load from any of the supported source shapes: an existing file or directory path (a tokenizer.json, or a directory containing one), a .tiktoken vocabulary file, a packaged encoding name (see Gigatoken::Encodings::NAMES, e.g. "cl100k_base"), or a HuggingFace Hub repo id like "openai-community/gpt2". A .tiktoken file carries no pretokenizer scheme of its own, so one must be named explicitly via pretokenizer: — nothing here is guessed. Packaged encoding names are checked before the Hub-repo-id shape: a bare name like "o200k_base" is also shaped like a legacy repo id, and must resolve locally rather than reach the network. Names the registry knows but doesn't package (see Encodings.unpackable_reason, e.g. "p50k_base") are intercepted here too, raising the same explanation from_encoding gives rather than reaching the Hub — but only those; an unrecognized bare name like "gpt2" still dispatches to the Hub.

Raises:



70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/gigatoken/tokenizer.rb', line 70

def self.load(source, pretokenizer: nil, special_tokens: {}, revision: "main", hub: Hub.new)
  source = source.to_s
  if source.end_with?(".tiktoken")
    unless pretokenizer
      raise Error, "#{source.inspect}: a .tiktoken file carries no pretokenizer scheme of its own — " \
        "pass pretokenizer: (one of #{Native.pretokenizer_names.join(", ")})"
    end
    return from_tiktoken(source, pretokenizer: pretokenizer, special_tokens: special_tokens)
  end
  return from_file(source) if File.exist?(source)
  return from_encoding(source) if Encodings::NAMES.include?(source) || Encodings.unpackable_reason(source)
  return from_hub(source, revision: revision, hub: hub) if Hub.looks_like_repo_id?(source)

  raise Error, "#{source.inspect}: no such file or directory, not a .tiktoken path, and doesn't look like a HuggingFace Hub repo id"
end

Instance Method Details

#cache_entries ⇒ Object

Cached pretoken/unit entries on this tokenizer's single-document encode path — see Gigatoken.max_cache_bytes.



148
149
150
 
# File 'lib/gigatoken/tokenizer.rb', line 148

def cache_entries
  @native.cache_entries
end

#decode(ids) ⇒ Object 



130
131
132
 
# File 'lib/gigatoken/tokenizer.rb', line 130

def decode(ids)
  @native.decode(ids)
end

#encode(text) ⇒ Object 



97
98
99
 
# File 'lib/gigatoken/tokenizer.rb', line 97

def encode(text)
  @native.encode(text)
end

#encode_batch(texts, packed: false) ⇒ Object

Returns a ragged Array of Arrays of token ids, one row per document — or, with packed: true, a Gigatoken::PackedResult (one IO::Buffer of token ids plus per-document lengths), avoiding the per-token Ruby array materialization the ragged shape costs.



105
106
107
108
109
110
111
 
# File 'lib/gigatoken/tokenizer.rb', line 105

def encode_batch(texts, packed: false)
  if packed
    PackedResult.new(*@native.encode_batch_packed(texts))
  else
    @native.encode_batch(texts)
  end
end

#encode_files(source, separator: nil, parallel: true, packed: false) ⇒ Object

Tokenize whole files in Rust: reads and encodes them in one fused pass without the documents ever becoming Ruby objects. source is a Native::Text,Jsonl,ParquetFileSource, a single path, or an array of paths; bare path(s) are wrapped in a TextFileSource (with separator, if given). Returns a ragged Array of Arrays of token ids, one row per document — or, with packed: true, a Gigatoken::PackedResult. parallel: false loads and encodes everything on the calling thread instead, with identical output, never touching the core worker pool.



121
122
123
124
125
126
127
128
 
# File 'lib/gigatoken/tokenizer.rb', line 121

def encode_files(source, separator: nil, parallel: true, packed: false)
  source = Native::TextFileSource.new(Array(source).map(&:to_s), separator: separator) unless FILE_SOURCE_CLASSES.any? { |klass| source.is_a?(klass) }
  if packed
    PackedResult.new(*@native.encode_files_packed(source, parallel: parallel))
  else
    @native.encode_files(source, parallel: parallel)
  end
end

#merges ⇒ Object 



142
143
144
 
# File 'lib/gigatoken/tokenizer.rb', line 142

def merges
  @native.merges
end

#vocab ⇒ Object 



138
139
140
 
# File 'lib/gigatoken/tokenizer.rb', line 138

def vocab
  @native.vocab
end

#vocab_size ⇒ Object 



134
135
136
 
# File 'lib/gigatoken/tokenizer.rb', line 134

def vocab_size
  @native.vocab_size
end