Module: Gigatoken

Defined in:
lib/gigatoken.rb,
lib/gigatoken/cli.rb,
lib/gigatoken/hub.rb,
lib/gigatoken/version.rb,
lib/gigatoken/cli/bench.rb,
lib/gigatoken/encodings.rb,
lib/gigatoken/tokenizer.rb,
lib/gigatoken/cli/support.rb,
lib/gigatoken/cli/validate.rb,
lib/gigatoken/packed_result.rb

Defined Under Namespace

Modules: CLI, Encodings Classes: Error, Hub, PackedResult, Tokenizer

Constant Summary collapse

NATIVE_EXTENSIONS = 
%w[.bundle .so .rb].freeze
VERSION = 
"0.2.1"

Class Method Summary collapse

Class Method Details

.locate_native(base, ruby_version: RUBY_VERSION) ⇒ Object

Precompiled native gems ship per-ABI subdirs (gigatoken/4.0/...), the source-gem rake compile build lands flat (gigatoken/...). Pick whichever exists for the current Ruby ABI, with the per-ABI path winning when both are present.



30
31
32
33
34
 
# File 'lib/gigatoken.rb', line 30

def self.locate_native(base, ruby_version: RUBY_VERSION)
  abi = ruby_version[/\d+\.\d+/]
  candidates = [File.join(base, abi, "gigatoken_rb"), File.join(base, "gigatoken_rb")]
  candidates.find { |stem| NATIVE_EXTENSIONS.any? { |ext| File.exist?(stem + ext) } }
end

.max_cache_bytes ⇒ Object

The process-global encode-cache budget in bytes per worker (a parallel batch encode may use up to workers x budget), applied to tokenizers of either backend constructed afterward — changing it never affects an already-built Tokenizer. nil means unbounded. Default: 512 MiB.



15
16
17
 
# File 'lib/gigatoken.rb', line 15

def max_cache_bytes
  Native.get_max_cache_bytes
end

.max_cache_bytes=(bytes) ⇒ Object 



19
20
21
 
# File 'lib/gigatoken.rb', line 19

def max_cache_bytes=(bytes)
  Native.set_max_cache_bytes(bytes)
end