Class: Ace::Test::EndToEndRunner::Organisms::TestOrchestrator

Inherits:
Object
  • Object
show all
Defined in:
lib/ace/test/end_to_end_runner/organisms/test_orchestrator.rb

Overview

Orchestrates E2E test discovery, execution, and reporting

Handles both single test and package-wide test execution flows. Coordinates between test discovery, scenario parsing, LLM execution, and report writing.

For CLI providers: agents write their own reports via workflow/skill, so the orchestrator skips ReportWriter and looks for agent-written report directories on disk.

For API providers: orchestrator writes reports as before.

Instance Method Summary collapse

Constructor Details

#initialize(provider: nil, timeout: nil, parallel: nil, base_dir: nil, timestamp_generator: nil, executor: nil, progress: false, discoverer: nil, integration_runner: nil, scenario_loader: nil, report_writer: nil, suite_report_writer: nil, setup_executor_factory: nil, runtime_builder: nil) ⇒ TestOrchestrator

Returns a new instance of TestOrchestrator.

Parameters:

  • provider (String) (defaults to: nil) —

    LLM provider:model string

  • timeout (Integer) (defaults to: nil) —

    Request timeout per test in seconds

  • parallel (Integer) (defaults to: nil) —

    Number of tests to run in parallel

  • base_dir (String) (defaults to: nil) —

    Base directory for test discovery

  • timestamp_generator (#call) (defaults to: nil) —

    Callable that returns a timestamp string

  • executor (#execute) (defaults to: nil) —

    Injectable test executor (for testing)

  • progress (Boolean) (defaults to: false) —

    Enable animated progress display



33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/ace/test/end_to_end_runner/organisms/test_orchestrator.rb', line 33

def initialize(provider: nil, timeout: nil, parallel: nil, base_dir: nil, timestamp_generator: nil,
  executor: nil, progress: false, discoverer: nil, integration_runner: nil,
  scenario_loader: nil, report_writer: nil, suite_report_writer: nil,
  setup_executor_factory: nil, runtime_builder: nil)
  config = Molecules::ConfigLoader.load
  @provider = provider || config.dig("execution", "runner_provider") ||
    config.dig("execution", "provider") || "claude:sonnet"
  @timeout = timeout || config.dig("execution", "timeout") || 300
  @parallel = parallel || config.dig("execution", "parallel") || 3
  @base_dir = base_dir || Dir.pwd
  @timestamp_generator = timestamp_generator || method(:default_timestamp)
  @progress = progress
  @discoverer = discoverer || Molecules::TestDiscoverer.new
  @integration_runner = integration_runner || Molecules::IntegrationRunner.new(base_dir: @base_dir)
  @loader = scenario_loader || Molecules::ScenarioLoader.new
  @executor = executor || Molecules::TestExecutor.new(provider: @provider, timeout: @timeout, config: config)
  @report_writer = report_writer || Molecules::ReportWriter.new
  @suite_report_writer = suite_report_writer || Molecules::SuiteReportWriter.new(config: config)
  @setup_executor_factory = setup_executor_factory || ->(sandbox_backend: nil) { Molecules::SetupExecutor.new(sandbox_backend: sandbox_backend) }
  @runtime_builder = runtime_builder || Molecules::SandboxRuntimeBuilder.new(
    source_root: @base_dir,
    ruby_version: config.dig("sandbox", "ruby_version") || Molecules::ConfigLoader.default_sandbox_ruby_version
  )
end

Instance Method Details

#run(package:, test_id: nil, test_cases: nil, verify: false, tags: nil, cli_args: nil, run_id: nil, report_dir: nil, output: $stdout) ⇒ Array<Models::TestResult>

Run E2E tests for a package, optionally filtering by test ID

Parameters:

  • package (String) —

    Package name (e.g., “ace-lint”)

  • test_id (String, nil) (defaults to: nil) —

    Optional test ID to run specific test

  • test_cases (Array<String>, nil) (defaults to: nil) —

    Optional normalized test case IDs to filter

  • tags (Array<String>, nil) (defaults to: nil) —

    Optional scenario tags for discovery filtering

  • cli_args (String, nil) (defaults to: nil) —

    Extra args for CLI providers

  • output (IO) (defaults to: $stdout) —

    Output stream for progress messages (default: $stdout)

Returns:



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/ace/test/end_to_end_runner/organisms/test_orchestrator.rb', line 67

def run(package:, test_id: nil, test_cases: nil, verify: false, tags: nil,
  cli_args: nil, run_id: nil, report_dir: nil, output: $stdout)
  integration_files = @discoverer.find_integration_tests(
    package: package,
    base_dir: @base_dir
  )

  # Discover tests
  files = @discoverer.find_tests(
    package: package,
    test_id: test_id,
    tags: tags,
    base_dir: @base_dir
  )

  if files.empty? && integration_files.empty?
    output.puts "No E2E tests found in #{package}" +
      (test_id ? " matching #{test_id}" : "")
    return []
  end

  # Generate timestamp for this run (use external run_id when provided)
  timestamp = run_id || generate_timestamp

  if files.size == 1 && integration_files.empty?
    run_single_test(
      files.first,
      timestamp,
      cli_args,
      output,
      test_cases: test_cases,
      verify: verify,
      report_dir: report_dir
    )
  else
    run_package_tests(
      files,
      package,
      timestamp,
      cli_args,
      output,
      test_cases: test_cases,
      verify: verify,
      integration_files: integration_files
    )
  end
end