Class: SkillBench::Services::RunnerService

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/services/runner_service.rb

Overview

Orchestrates the execution of an eval with baseline and context runs. Coordinates multiple services to resolve entities, spawn agents, and evaluate results.

Defined Under Namespace

Classes: EvaluationContext

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(eval_name:, skill_names:, pack: nil, registry_manifest: nil) ⇒ RunnerService

Returns a new instance of RunnerService.

Parameters:

  • eval_name (String) —

    Name or path of the eval

  • skill_names (Array<String>) —

    Names of the skills

  • pack (String, nil) (defaults to: nil) —

    Optional pack name

  • registry_manifest (String, nil) (defaults to: nil) —

    Optional registry.json path



44
45
46
47
48
49
 
# File 'lib/skill_bench/services/runner_service.rb', line 44

def initialize(eval_name:, skill_names:, pack: nil, registry_manifest: nil)
  @eval_name = eval_name
  @skill_names = skill_names
  @pack = pack
  @registry_manifest = registry_manifest
end

Class Method Details

.call(eval_name:, skill_names:, pack: nil, registry_manifest: nil) ⇒ Hash

Runs an eval with the given parameters.

Parameters:

  • eval_name (String) —

    Name or path of the eval to run

  • skill_names (Array<String>) —

    Names of the skills to use

  • pack (String, nil) (defaults to: nil) —

    Optional pack name for registry-based skill resolution

  • registry_manifest (String, nil) (defaults to: nil) —

    Optional path to registry.json manifest

Returns:

  • (Hash) —

    Result from EvaluationRunner



31
32
33
34
35
36
37
38
 
# File 'lib/skill_bench/services/runner_service.rb', line 31

def self.call(eval_name:, skill_names:, pack: nil, registry_manifest: nil)
  new(
    eval_name: eval_name,
    skill_names: skill_names,
    pack: pack,
    registry_manifest: registry_manifest
  ).call
end

Instance Method Details

#call ⇒ Hash

Executes the eval: resolves entities, runs baseline and context, evaluates.

Returns:

  • (Hash) —

    Evaluation result with deltas and verdict.

Raises:

  • (Errno::ENOENT) —

    when the eval directory does not exist.

  • (ArgumentError) —

    when a skill cannot be resolved.



56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/skill_bench/services/runner_service.rb', line 56

def call
  evaluation = EvalResolver.call(eval_name)
  skills = SkillResolverService.call(skill_names, pack: pack, registry_manifest: registry_manifest)
  provider_result = ProviderResolver.call

  return config_error_result(provider_result[:error], evaluation, provider_result[:provider]) unless provider_result[:success]

  provider = provider_result[:provider]
  config = provider_result[:config]

  skill_context = ContextLoaderService.call(skills)
  return empty_context_error_result(evaluation, provider) if skill_context.strip.empty?

  baseline_output, context_output = run_agents_concurrently(evaluation, skills, skill_context, provider, config)
  return agent_error_result(baseline_output, 'baseline', evaluation, provider) if baseline_output[:status] == :error
  return agent_error_result(context_output, 'context', evaluation, provider) if context_output[:status] == :error

  context = EvaluationContext.new(
    evaluation: evaluation,
    skill_context: skill_context,
    baseline_output: baseline_output,
    context_output: context_output,
    provider: provider,
    config: config
  )
  evaluate_and_record_trend(context)
end