Class: Ace::Test::EndToEndRunner::Molecules::TestExecutor

Inherits:
Object
  • Object
show all
Defined in:
lib/ace/test/end_to_end_runner/molecules/test_executor.rb

Overview

Executes a single E2E test scenario via LLM

Routes execution through two paths based on provider type:

  • CLI providers (claude, gemini, codex): deterministic standalone pipeline

  • API providers (google, anthropic): prompt-based prediction mode

Instance Method Summary collapse

Constructor Details

#initialize(provider: nil, timeout: nil, config: nil, sandbox_backend_factory: nil) ⇒ TestExecutor

Returns a new instance of TestExecutor.

Parameters:

  • provider (String) (defaults to: nil) —

    LLM provider:model string

  • timeout (Integer) (defaults to: nil) —

    Request timeout in seconds

  • config (Hash) (defaults to: nil) —

    Configuration hash (string keys) from ConfigLoader



19
20
21
22
23
24
25
26
27
28
29
30
31
 
# File 'lib/ace/test/end_to_end_runner/molecules/test_executor.rb', line 19

def initialize(provider: nil, timeout: nil, config: nil, sandbox_backend_factory: nil)
  config ||= Molecules::ConfigLoader.load
  @provider = provider || config.dig("execution", "runner_provider") ||
    config.dig("execution", "provider") || "claude:sonnet"
  @verifier_provider = config.dig("execution", "verifier_provider") ||
    config.dig("execution", "provider") || @provider
  @timeout = timeout || config.dig("execution", "timeout") || 300
  @prompt_builder = Atoms::PromptBuilder.new
  @cli_provider_adapter = Atoms::CliProviderAdapter.new(config)
  @sandbox_backend_factory = sandbox_backend_factory || lambda { |sandbox_path, source_root: nil|
    Molecules::BwrapSandboxBackend.new(sandbox_root: sandbox_path, source_root: source_root)
  }
end

Instance Method Details

#execute(scenario, cli_args: nil, run_id: nil, test_cases: nil, sandbox_path: nil, env_vars: nil, report_dir: nil, timeout: nil, verify: false) ⇒ Models::TestResult

Execute a single test scenario via LLM

Parameters:

  • scenario (Models::TestScenario) —

    The test scenario to execute

  • cli_args (String, nil) (defaults to: nil) —

    Extra args for CLI providers

  • run_id (String, nil) (defaults to: nil) —

    Pre-generated run ID for deterministic report paths

  • test_cases (Array<String>, nil) (defaults to: nil) —

    Optional test case IDs to filter

  • sandbox_path (String, nil) (defaults to: nil) —

    Path to pre-populated sandbox (skips LLM setup)

  • env_vars (Hash, nil) (defaults to: nil) —

    Environment variables from setup execution

  • report_dir (String, nil) (defaults to: nil) —

    Explicit report directory path (overrides computed path)

Returns:



43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/ace/test/end_to_end_runner/molecules/test_executor.rb', line 43

def execute(scenario, cli_args: nil, run_id: nil, test_cases: nil, sandbox_path: nil,
  env_vars: nil, report_dir: nil, timeout: nil, verify: false)
  resolved_timeout = timeout || @timeout
  if Atoms::CliProviderAdapter.cli_provider?(@provider)
    execute_via_pipeline(
      scenario,
      cli_args: cli_args,
      run_id: run_id,
      test_cases: test_cases,
      sandbox_path: sandbox_path,
      env_vars: env_vars,
      report_dir: report_dir,
      timeout: resolved_timeout
    )
  else
    execute_via_prompt(scenario, cli_args: cli_args, test_cases: test_cases, timeout: resolved_timeout)
  end
end

#execute_tc(test_case:, sandbox_path:, scenario:, cli_args: nil, run_id: nil, env_vars: nil) ⇒ Models::TestResult

Execute a single test case via LLM in a pre-populated sandbox

Parameters:

  • test_case (Models::TestCase) —

    The single test case to execute

  • sandbox_path (String) —

    Path to the pre-populated sandbox

  • scenario (Models::TestScenario) —

    The parent scenario for metadata

  • cli_args (String, nil) (defaults to: nil) —

    Extra args for CLI providers

  • run_id (String, nil) (defaults to: nil) —

    Pre-generated run ID

  • env_vars (Hash, nil) (defaults to: nil) —

    Environment variables from setup execution

Returns:



71
72
73
74
75
76
77
 
# File 'lib/ace/test/end_to_end_runner/molecules/test_executor.rb', line 71

def execute_tc(test_case:, sandbox_path:, scenario:, cli_args: nil, run_id: nil, env_vars: nil)
  if Atoms::CliProviderAdapter.cli_provider?(@provider)
    execute_tc_via_skill(test_case, sandbox_path, scenario, cli_args: cli_args, run_id: run_id, env_vars: env_vars)
  else
    execute_tc_via_prompt(test_case, sandbox_path, scenario, cli_args: cli_args)
  end
end