Class: SkillBench::Evaluation::Runner

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/evaluation/runner.rb

Overview

Orchestrates the evaluation pipeline.

Coordinates blind judging of baseline and context agent outputs, then computes deltas and determines the final verdict.

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {}) ⇒ Runner

Returns a new instance of Runner.

Parameters:

  • task (String) —

    The task description.

  • criteria (SkillBench::Criteria) —

    The eval criteria.

  • skill_context (String) —

    The skill context XML.

  • baseline_output (String) —

    The baseline agent output.

  • context_output (String) —

    The context agent output.

  • judge_params (Hash) (defaults to: {}) —

    Provider config passed to the Judge as client_params.



31
32
33
34
35
36
37
38
 
# File 'lib/skill_bench/evaluation/runner.rb', line 31

def initialize(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {})
  @task = task
  @criteria = criteria
  @skill_context = skill_context
  @baseline_output = baseline_output
  @context_output = context_output
  @judge_params = judge_params.is_a?(Hash) ? judge_params : {}
end

Class Method Details

.call(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {}) ⇒ Hash

Runs the evaluation pipeline.

Parameters:

  • task (String) —

    The task description.

  • criteria (SkillBench::Criteria) —

    The eval criteria.

  • skill_context (String) —

    The skill context XML.

  • baseline_output (String) —

    The baseline agent output.

  • context_output (String) —

    The context agent output.

  • judge_params (Hash) (defaults to: {}) —

    Provider config passed to the Judge as client_params (api_key, model, provider).

Returns:

  • (Hash) —

    Service response with report or error.



21
22
23
 
# File 'lib/skill_bench/evaluation/runner.rb', line 21

def self.call(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {})
  new(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params:).call
end

Instance Method Details

#call ⇒ Hash

Orchestrates judging and delta computation.

Returns:

  • (Hash) —

    Service response with report or error.



43
44
45
46
47
48
49
50
51
52
 
# File 'lib/skill_bench/evaluation/runner.rb', line 43

def call
  baseline_judge, context_judge = run_judges_concurrently
  return baseline_judge unless baseline_judge[:success]
  return context_judge unless context_judge[:success]

  compute_deltas(baseline_judge, context_judge)
rescue StandardError => e
  SkillBench::ErrorLogger.log_error(e, 'Evaluation::Runner Error')
  { success: false, response: { error: { message: e.message } } }
end