Class: SkillBench::Services::BatchRunnerService

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/services/batch_runner_service.rb

Overview

Orchestrates running many evals in a single batch.

Discovers every eval under a target directory and runs RunnerService over each, returning an aggregate envelope with per-eval results and a pass/fail summary.

Discovery reuses Runner.discover_task_dirs but never routes through the deprecated Task::Evaluator: each eval is executed by the supported RunnerService.

Constant Summary collapse

DEFAULT_EVALS_DIR =

Default directory scanned for evals when none is supplied.

'evals'
DEFAULT_THREADS =

Default batch-level thread count.

Each RunnerService.call already runs its baseline and context agents concurrently (#26), so this is kept modest to bound nested thread usage (batch threads x per-eval threads).

2

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(skill_names:, evals_dir:, pack:, registry_manifest:, threads:) ⇒ BatchRunnerService

Returns a new instance of BatchRunnerService.

Parameters:

  • skill_names (Array<String>) —

    Names of the skills

  • evals_dir (String) —

    Directory to scan for evals

  • pack (String, nil) —

    Optional pack name

  • registry_manifest (String, nil) —

    Optional registry.json path

  • threads (Integer) —

    Batch-level thread count



55
56
57
58
59
60
61
 
# File 'lib/skill_bench/services/batch_runner_service.rb', line 55

def initialize(skill_names:, evals_dir:, pack:, registry_manifest:, threads:)
  @skill_names = skill_names
  @evals_dir = evals_dir
  @pack = pack
  @registry_manifest = registry_manifest
  @threads = threads
end

Class Method Details

.call(skill_names:, evals_dir: DEFAULT_EVALS_DIR, pack: nil, registry_manifest: nil, threads: DEFAULT_THREADS) ⇒ Hash

Runs every eval discovered under evals_dir.

Parameters:

  • skill_names (Array<String>) —

    Names of the skills to apply to every eval

  • evals_dir (String) (defaults to: DEFAULT_EVALS_DIR) —

    Directory to scan for evals

  • pack (String, nil) (defaults to: nil) —

    Optional pack name for registry-based skill resolution

  • registry_manifest (String, nil) (defaults to: nil) —

    Optional path to registry.json manifest

  • threads (Integer) (defaults to: DEFAULT_THREADS) —

    Batch-level thread count

Returns:

  • (Hash) —

    Aggregate envelope with :results and :summary

Raises:

  • (ArgumentError) —

    when no evals are found under evals_dir



40
41
42
43
44
45
46
47
48
 
# File 'lib/skill_bench/services/batch_runner_service.rb', line 40

def self.call(skill_names:, evals_dir: DEFAULT_EVALS_DIR, pack: nil, registry_manifest: nil, threads: DEFAULT_THREADS)
  new(
    skill_names: skill_names,
    evals_dir: evals_dir,
    pack: pack,
    registry_manifest: registry_manifest,
    threads: threads
  ).call
end

Instance Method Details

#call ⇒ Hash

Discovers the target evals and runs each through RunnerService.

Returns:

  • (Hash) —

    Aggregate envelope with :results and :summary

Raises:

  • (ArgumentError) —

    when no evals are found under the directory



67
68
69
70
71
72
73
 
# File 'lib/skill_bench/services/batch_runner_service.rb', line 67

def call
  eval_dirs = discover_eval_dirs
  raise ArgumentError, "No evals found under #{evals_dir}" if eval_dirs.empty?

  results = run_all(eval_dirs)
  { results: results, summary: summarize(results) }
end