Class: LLMExperiment::Sanitizer

Inherits:
Object
  • Object
show all
Defined in:
lib/llm_experiment/sanitizer.rb

Overview

The gate between results-raw/ (gitignored) and results/ (committed).

A subject app may be private, so its raw transcripts cannot be published: a transcript contains whatever source the agent read. For those apps only derived measurements cross. An app marked publish_transcripts: true in experiment.yml crosses whole.

The check is deliberately a hard failure rather than a redaction. Something unexpected in a transcript should stop the commit and be looked at, not be quietly rewritten into something that looks safe.

Constant Summary collapse

FORBIDDEN =

Anything matching these must never reach the committed tree, whatever the app's privacy setting. Host paths identify the machine; the rest are secrets.

{
  "host home path" => %r{/Users/[a-z]},
  "AWS-style key" => /AKIA[0-9A-Z]{16}/,
  "private key block" => /-----BEGIN [A-Z ]*PRIVATE KEY-----/,
  "bearer token" => /\b(?:sk|pk)-[A-Za-z0-9_-]{20,}/,
  "rails master key" => /\b[0-9a-f]{32}\b(?=.*master)/i,
  "generic api secret" => /(?:api[_-]?key|secret|password|token)["'\s:=]+[A-Za-z0-9_-]{24,}/i
}.freeze
DERIVED =

Derived measurements always cross. Raw transcripts only for public apps.

%w[meta.json metrics.json prompt.txt].freeze
RAW = 
%w[transcript.jsonl events.jsonl agent.diff test_before.txt test_after.txt].freeze
PRIVATE_README = 
<<~MD
  Private application. Raw transcripts and diffs stay in `results-raw/`,
  which is gitignored; only the measurements derived from them are here.

  - `meta.json` what the trial did and whether the fix held
  - `metrics.json` the numbers `llmx metrics` reads
  - `prompt.txt` the exact prompt, which contains no application source
MD

Instance Method Summary collapse

Constructor Details

#initialize(experiment:) ⇒ Sanitizer

Returns a new instance of Sanitizer.



42
43
44
 
# File 'lib/llm_experiment/sanitizer.rb', line 42

def initialize(experiment:)
  @experiment = experiment
end

Instance Method Details

#check ⇒ Object

Report what would be refused, copy nothing.



51
 
# File 'lib/llm_experiment/sanitizer.rb', line 51

def check = run(check_only: true)

#public_apps ⇒ Object 



46
47
48
 
# File 'lib/llm_experiment/sanitizer.rb', line 46

def public_apps
  @public_apps ||= @experiment.apps.values.select(&:publish_transcripts).map(&:key)
end

#publish ⇒ Object

Copy what may cross, or refuse and leave nothing behind.



54
 
# File 'lib/llm_experiment/sanitizer.rb', line 54

def publish = run(check_only: false)