Class: RubyLLM::Contract::RakeTask::SuiteGate

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby_llm/contract/rake_task/suite_gate.rb

Overview

Encapsulates the pass/fail gate that runs after ‘RakeTask#define_task` has collected eval reports. Extracted from the prior `define_task` god-method so each gating dimension (cost, score, regression) is testable in isolation.

Returns a ‘Verdict` value object with:

- `passed?`         — overall gate verdict
- `abort_reason`    — String for `abort` when `passed? == false`, nil otherwise
- `passed_reports`  — [[host, report], ...] of reports that individually passed
                      (used to decide which baselines to save)
- `suite_cost`      — total cost across all reports

Gate ordering (preserved from pre-refactor behaviour):

1. cost gate runs FIRST — if `maximum_cost` set and exceeded, the
   suite aborts before any score check; passed_reports is empty.
2. score gate runs per-report; a report passes if
   `report_meets_score?` AND `!check_regression`.
3. overall passed = ALL reports passed AND cost gate not tripped.

Defined Under Namespace

Classes: Verdict

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:) ⇒ SuiteGate

Returns a new instance of SuiteGate.



40
41
42
43
44
45
46
 
# File 'lib/ruby_llm/contract/rake_task/suite_gate.rb', line 40

def initialize(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:)
  @host_reports = host_reports
  @minimum_score = minimum_score
  @maximum_cost = maximum_cost
  @fail_on_regression = fail_on_regression
  @verdict = build_verdict
end

Instance Attribute Details

#verdict ⇒ Object (readonly)

Returns the value of attribute verdict.



38
39
40
 
# File 'lib/ruby_llm/contract/rake_task/suite_gate.rb', line 38

def verdict
  @verdict
end

Class Method Details

.evaluate(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:) ⇒ Object 



31
32
33
34
35
36
 
# File 'lib/ruby_llm/contract/rake_task/suite_gate.rb', line 31

def self.evaluate(host_reports:, minimum_score:, maximum_cost:, fail_on_regression:)
  new(host_reports: host_reports,
      minimum_score: minimum_score,
      maximum_cost: maximum_cost,
      fail_on_regression: fail_on_regression).verdict
end