Class: SkillBench::Services::ComparisonRunner

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/services/comparison_runner.rb

Overview

Runs both variants of a skill comparison.

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(variant_a, variant_b, skill_name, eval_path, manifest_path: nil) ⇒ ComparisonRunner

Returns a new instance of ComparisonRunner.

Parameters:

  • variant_a (Hash) —

    First variant specification

  • variant_b (Hash) —

    Second variant specification

  • skill_name (String) —

    Name of the skill to compare

  • eval_path (String) —

    Path to the eval directory

  • manifest_path (String, nil) (defaults to: nil) —

    Optional path to registry manifest



27
28
29
30
31
32
33
 
# File 'lib/skill_bench/services/comparison_runner.rb', line 27

def initialize(variant_a, variant_b, skill_name, eval_path, manifest_path: nil)
  @variant_a = variant_a
  @variant_b = variant_b
  @skill_name = skill_name
  @eval_path = eval_path
  @manifest_path = manifest_path
end

Class Method Details

.call(variant_a, variant_b, skill_name, eval_path, manifest_path: nil) ⇒ Hash

Runs both variants and returns their results.

Parameters:

  • variant_a (Hash) —

    First variant specification

  • variant_b (Hash) —

    Second variant specification

  • skill_name (String) —

    Name of the skill to compare

  • eval_path (String) —

    Path to the eval directory

  • manifest_path (String, nil) (defaults to: nil) —

    Optional path to registry manifest

Returns:

  • (Hash) —

    Hash with :result_a and :result_b keys



18
19
20
 
# File 'lib/skill_bench/services/comparison_runner.rb', line 18

def self.call(variant_a, variant_b, skill_name, eval_path, manifest_path: nil)
  new(variant_a, variant_b, skill_name, eval_path, manifest_path: manifest_path).call
end

Instance Method Details

#call ⇒ Hash

Runs both variants and returns their results.

Returns:

  • (Hash) —

    Hash with :result_a and :result_b keys



38
39
40
41
42
43
44
45
46
 
# File 'lib/skill_bench/services/comparison_runner.rb', line 38

def call
  skill_paths_a = VariantResolver.call(@variant_a, @skill_name, manifest_path: @manifest_path)
  skill_paths_b = VariantResolver.call(@variant_b, @skill_name, manifest_path: @manifest_path)

  result_a = RunnerService.call(eval_name: @eval_path, skill_names: skill_paths_a)
  result_b = RunnerService.call(eval_name: @eval_path, skill_names: skill_paths_b)

  { result_a: result_a, result_b: result_b }
end