Class: Gigatoken::CLI::Validate

Inherits:
Dry::CLI::Command
  • Object
show all
Defined in:
lib/gigatoken/cli/validate.rb

Overview

gigatoken validate TOKENIZER FILES... — a Ruby-vs-Ruby consistency check: encode FILES via Tokenizer#encode_files (native file loading and splitting) and via a Ruby-side split plus Tokenizer#encode_batch, and confirm the two paths agree per document. Cross-library validation against another tokenizer implementation is out of scope for v1 (BRIEF §3.4).

Instance Method Summary collapse

Instance Method Details

#call(tokenizer:, files:, doc_separator: nil, pretokenizer: nil) ⇒ Object 



22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/gigatoken/cli/validate.rb', line 22

def call(tokenizer:, files:, doc_separator: nil, pretokenizer: nil, **)
  gt_tokenizer = Support.load_tokenizer(tokenizer, pretokenizer: pretokenizer)

  via_files = gt_tokenizer.encode_files(Support.text_file_source(files, doc_separator))
  via_batch = gt_tokenizer.encode_batch(Support.split_docs(files, doc_separator))

  if via_files.length != via_batch.length
    err.puts "validation FAILED: encode_files produced #{via_files.length} documents, encode_batch produced #{via_batch.length}"
    exit(1)
  end

  via_files.each_with_index do |doc, index|
    next if doc == via_batch[index]

    report_mismatch(index, doc, via_batch[index])
    exit(1)
  end

  out.puts "validation OK: #{via_files.length} documents match"
rescue Gigatoken::Error => e
  err.puts "error: #{e.message}"
  exit(1)
end