Class: Ace::Test::EndToEndRunner::Atoms::CliProviderAdapter

Inherits:
Object
  • Object
show all
Defined in:
lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb

Overview

Holds CLI-provider detection and CLI-args helpers.

Standalone scenario execution for CLI providers now runs through the deterministic runner/verifier pipeline. Provider lists and CLI args are configurable via config.yml.

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(config = {}) ⇒ CliProviderAdapter

Returns a new instance of CliProviderAdapter.

Parameters:

  • config (Hash) (defaults to: {}) —

    Configuration hash (string keys) with providers section



14
15
16
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 14

def initialize(config = {})
  @cli_providers = config.dig("providers", "cli") || %w[claude gemini codex codexoss opencode pi]
end

Class Method Details

.cli_provider?(provider_string) ⇒ Boolean

Check if a provider string refers to a CLI provider

Parameters:

  • provider_string (String) —

    Provider:model string (e.g., “claude:sonnet”)

Returns:

  • (Boolean) 


22
23
24
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 22

def self.cli_provider?(provider_string)
  default_instance.cli_provider?(provider_string)
end

.default_instance ⇒ CliProviderAdapter

Lazily-loaded default instance backed by ConfigLoader

Returns:



165
166
167
168
169
170
171
172
173
174
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 165

def self.default_instance
  @default_instance ||= begin
    config = if defined?(Molecules::ConfigLoader)
      Molecules::ConfigLoader.load
    else
      {}
    end
    new(config)
  end
end

.provider_name(provider_string) ⇒ String

Extract provider name from provider:model string

Parameters:

  • provider_string (String) —

    e.g., “claude:sonnet”

Returns:

  • (String) —

    e.g., “claude”



30
31
32
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 30

def self.provider_name(provider_string)
  provider_string.to_s.split(":").first.to_s
end

.reset_default_instance! ⇒ Object

Reset the default instance (for testing)



177
178
179
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 177

def self.reset_default_instance!
  @default_instance = nil
end

Instance Method Details

#build_execution_prompt(command:, tc_mode:) ⇒ Object 



45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 45

def build_execution_prompt(command:, tc_mode:)
  return_contract = if tc_mode
    "- **Test ID**: ...\n- **TC ID**: ...\n- **Status**: pass | fail\n- **Report Paths**: ...\n- **Observations**: ...\n- **Issues**: ... (optional legacy alias)"
  else
    "- **Test ID**: ...\n- **Status**: pass | fail | partial\n- **Passed**: ...\n- **Failed**: ...\n- **Total**: ...\n- **Report Paths**: ...\n- **Observations**: ...\n- **Issues**: ... (optional legacy alias)"
  end

  <<~PROMPT.strip
    Run this as a slash command in the agent chat interface (not in bash):
    #{command}

    Execution requirements:
    - Do not run `/ace-...` inside a shell command.
    - If slash commands are unavailable, stop and report that limitation in `Observations`.
    - Write reports under `.ace-local/test-e2e/*-reports/`.
    - `Observations` is required and must be a concise factual summary of actions, outcomes, and blockers without verdict language.
    - Return only this structured summary:
    #{return_contract}
  PROMPT
end

#build_skill_prompt(scenario, run_id: nil, test_cases: nil, sandbox_path: nil, env_vars: nil, report_dir: nil) ⇒ String

Build a skill invocation prompt for scenario-level execution

Parameters:

  • scenario (Models::TestScenario) —

    The test scenario

  • run_id (String, nil) (defaults to: nil) —

    Pre-generated run ID for deterministic report paths

  • test_cases (Array<String>, nil) (defaults to: nil) —

    Optional test case IDs to filter

  • sandbox_path (String, nil) (defaults to: nil) —

    Path to pre-populated sandbox (skips setup steps)

  • env_vars (Hash, nil) (defaults to: nil) —

    Environment variables from setup execution

  • report_dir (String, nil) (defaults to: nil) —

    Explicit report directory path (overrides computed path)

Returns:

  • (String) —

    Skill invocation prompt



77
78
79
80
81
82
83
84
85
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 77

def build_skill_prompt(scenario, run_id: nil, test_cases: nil, sandbox_path: nil, env_vars: nil, report_dir: nil)
  cmd = "/as-e2e-run #{scenario.package} #{scenario.test_id}"
  cmd += " #{test_cases.join(",")}" if test_cases&.any?
  cmd += " --run-id #{run_id}" if run_id
  cmd += " --sandbox #{sandbox_path}" if sandbox_path
  cmd += " --env #{env_vars.map { |k, v| "#{k}=#{v}" }.join(",")}" if env_vars&.any?
  cmd += " --report-dir #{report_dir}" if report_dir
  build_execution_prompt(command: cmd, tc_mode: false)
end

#build_tc_skill_prompt(test_case:, scenario:, sandbox_path:, run_id: nil, env_vars: nil) ⇒ String

Build a TC-level skill invocation prompt

Parameters:

  • test_case (Models::TestCase) —

    The single test case

  • scenario (Models::TestScenario) —

    The parent scenario

  • sandbox_path (String) —

    Path to the pre-populated sandbox

  • run_id (String, nil) (defaults to: nil) —

    Pre-generated run ID

  • env_vars (Hash, nil) (defaults to: nil) —

    Environment variables from setup execution

Returns:

  • (String) —

    Skill invocation prompt



95
96
97
98
99
100
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 95

def build_tc_skill_prompt(test_case:, scenario:, sandbox_path:, run_id: nil, env_vars: nil)
  cmd = "/as-e2e-run #{scenario.package} #{scenario.test_id} #{test_case.tc_id} --tc-mode --sandbox #{sandbox_path}"
  cmd += " --run-id #{run_id}" if run_id
  cmd += " --env #{env_vars.map { |k, v| "#{k}=#{v}" }.join(",")}" if env_vars&.any?
  build_execution_prompt(command: cmd, tc_mode: true)
end

#build_verifier_prompt(scenario, run_id: nil, sandbox_path: nil, test_cases: nil, report_dir: nil) ⇒ Object

Build an independent verifier prompt.

This is intentionally a second invocation to avoid sharing runner context.



105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 105

def build_verifier_prompt(scenario, run_id: nil, sandbox_path: nil, test_cases: nil, report_dir: nil)
  report_dir ||= if run_id
    ".ace-local/test-e2e/#{scenario.dir_name(run_id)}-reports"
  end

  tc_filter = test_cases&.any? ? test_cases.join(", ") : "all discovered test cases"
  sandbox_info = sandbox_path || "(unknown)"
  report_info = report_dir || "(unknown)"

  <<~PROMPT.strip
    You are the independent verifier for an E2E scenario.

    Verify this scenario in a new, isolated agent context:
    - Package: #{scenario.package}
    - Test ID: #{scenario.test_id}
    - Sandbox path: #{sandbox_info}
    - Report directory: #{report_info}
    - Scope: #{tc_filter}

    Verification requirements:
    - Inspect sandbox artifacts and scenario files directly.
    - Judge from sandbox state first, then runner observations, then raw debug captures only when needed.
    - Evaluate each test case using `TC-*.verify.md` criteria when present.
    - Classify each failed test case with one category:
      `test-spec-error`, `tool-bug`, `runner-error`, or `infrastructure-error`.
    - Write/update report files under the report directory.
    - Use TC-first schema in report frontmatter and metadata.

    Return only this structured summary:
    - **Test ID**: ...
    - **Status**: pass | fail | partial | error
    - **TCs Passed**: ...
    - **TCs Failed**: ...
    - **TCs Total**: ...
    - **Score**: ...
    - **Verdict**: pass | partial | fail
    - **Failed TCs**: TC-001:tool-bug, TC-002:runner-error (or `None`)
    - **Issues**: ...
  PROMPT
end

#cli_provider?(provider_string) ⇒ Boolean

Instance method: check if a provider string refers to a CLI provider

Resolves role: references to their concrete provider before checking.

Parameters:

  • provider_string (String) —

    Provider:model string (e.g., “claude:sonnet”, “role:e2e-runner”)

Returns:

  • (Boolean) 


40
41
42
43
 
# File 'lib/ace/test/end_to_end_runner/atoms/skill_prompt_builder.rb', line 40

def cli_provider?(provider_string)
  resolved = resolve_provider_name(provider_string)
  @cli_providers.include?(resolved)
end