Module: Gigatoken::CLI::Support

Defined in:
lib/gigatoken/cli/support.rb

Overview

Helpers shared by the bench and validate commands: tokenizer loading, byte-size parsing, document splitting, and CPU identification.

Class Method Summary collapse

Class Method Details

.cpu_info ⇒ Object

The benchmark machine's CPU as "name, N cores", plus ", M sockets" when there is more than one socket.



78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/gigatoken/cli/support.rb', line 78

def cpu_info
  name, cores, sockets =
    case RbConfig::CONFIG["host_os"]
    when /darwin/ then darwin_cpu_info
    when /linux/ then linux_cpu_info
    end
  name ||= RbConfig::CONFIG["host_cpu"] || "unknown CPU"
  cores ||= Etc.nprocessors
  parts = [name, "#{cores} core#{"s" unless cores == 1}"]
  parts << "#{sockets} sockets" if sockets && sockets > 1
  parts.join(", ")
end

.load_tokenizer(spec, pretokenizer: nil) ⇒ Object

Load TOKENIZER: a tokenizer.json path/directory, a packaged tiktoken encoding name, a HuggingFace repo id, or a .tiktoken file — see Gigatoken::Tokenizer.load. pretokenizer: is forwarded as-is; it's required for a bare .tiktoken path (which carries no scheme of its own) and ignored for the other shapes.



22
23
24
 
# File 'lib/gigatoken/cli/support.rb', line 22

def load_tokenizer(spec, pretokenizer: nil)
  Gigatoken::Tokenizer.load(spec, pretokenizer: pretokenizer)
end

.parse_size(text) ⇒ Object

Parse a decimal byte size like "100MB", "2.5GB", or "1000000"; "none"/"unlimited" means no limit.

Raises:



28
29
30
31
32
33
34
35
 
# File 'lib/gigatoken/cli/support.rb', line 28

def parse_size(text)
  return nil if ["none", "unlimited"].include?(text.strip.downcase)

  match = SIZE_PATTERN.match(text)
  raise Gigatoken::Error, "cannot parse size #{text.inspect}; expected something like 100MB (or 'none')" unless match

  (match[1].to_f * SIZE_UNITS.fetch(match[2].downcase)).to_i
end

.split_docs(files, separator) ⇒ Object

Whole files as raw bytes, one document per file, or (with a separator) the separator-split pieces of each file in order, empty documents skipped.



46
47
48
49
50
51
52
 
# File 'lib/gigatoken/cli/support.rb', line 46

def split_docs(files, separator)
  raws = files.map { |file| File.binread(file.to_s) }
  return raws if separator.nil?

  sep = separator.b
  raws.flat_map { |raw| raw.split(sep).reject(&:empty?) }
end

.subset_docs(docs, limit_bytes) ⇒ Object

The prefix of docs totalling at most limit_bytes, byte- truncating the final document to fill the budget. Unlike a text-comparison tool, gigatoken encodes raw bytes and does not require the cut to land on a UTF-8 character boundary.



58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/gigatoken/cli/support.rb', line 58

def subset_docs(docs, limit_bytes)
  return docs if limit_bytes.nil?

  subset = []
  used = 0
  docs.each do |doc|
    room = limit_bytes - used
    if doc.bytesize <= room
      subset << doc
      used += doc.bytesize
    else
      subset << doc.byteslice(0, room) if room > 0
      break
    end
  end
  subset
end

.text_file_source(files, separator) ⇒ Object

A Native::TextFileSource for FILES, split on separator when given.



39
40
41
 
# File 'lib/gigatoken/cli/support.rb', line 39

def text_file_source(files, separator)
  Gigatoken::Native::TextFileSource.new(files.map(&:to_s), separator: separator)
end