Class: Ace::Test::EndToEndRunner::Molecules::PipelinePromptBundler

Inherits:
Object
  • Object
show all
Defined in:
lib/ace/test/end_to_end_runner/molecules/pipeline_prompt_bundler.rb

Overview

Prepares deterministic runner/verifier prompt files for pipeline execution.

Constant Summary collapse

RUNNER_SYSTEM_PROMPT = 
<<~PROMPT
  You are an E2E test executor working in a sandbox directory.

  Rules:
  - Execute each goal in order
  - Treat the initial working directory as SANDBOX_ROOT; if a goal needs commands in a created worktree, cd there for execution but keep any declared outcome artifacts under SANDBOX_ROOT/results
  - Preserve the sandbox runtime environment; do not reset PATH, HOME, or other provided env vars
  - If `ACE_E2E_SANDBOX_RUNTIME_ROOT` is set, make sure command execution uses `$ACE_E2E_SANDBOX_RUNTIME_ROOT/bin` on PATH in the shell where you run scenario commands
  - Run `ace-*` commands directly; do not wrap them with `timeout`, `env -i`, or other execution wrappers that can change behavior or hide diagnostics
  - Do not bypass the public CLI with repo-local executables such as `./exe/ace-*`, `bin/ace-*`, or `ruby .../exe/ace-*`
  - Do not fabricate output - all artifacts must come from real tool execution
  - Never background commands or start dependent verification captures before the command they verify has completed
  - When a goal requires command captures, keep stdout and stderr separate; do not merge streams and do not use `2>&1`
  - A command capture set is incomplete unless the matching `.stdout`, `.stderr`, and `.exit` files all exist
  - Persist each command's `.stdout`, `.stderr`, and `.exit` files immediately after that command finishes, before starting the next command
  - For commands that establish state, write that command's `.exit` file before running any list/status/fs-check/tmux verification for the same goal
  - When a successful command prints a filesystem path to a generated artifact, copy that artifact into `results/` if the goal asks for supporting evidence from the generated file
  - If a goal fails, note the failure and continue to the next goal
  - Do not create synthetic helper reports or temp input files under results/ unless the scenario explicitly treats them as product outcomes
  - After all goals, return concise runner observations describing what you did and what happened
PROMPT
VERIFIER_SYSTEM_PROMPT = 
<<~PROMPT
  You are an E2E test verifier. You inspect artifacts and render PASS/FAIL verdicts.

  Rules:
  - Evaluate each goal independently based on sandbox state first, then runner observations, then raw debug captures only when needed
  - Treat declared artifacts and helper filenames as hints, not as the source of truth
  - If a helper file is missing or stale, inspect the sandbox directly before failing the goal
  - Use artifact mtimes to detect runner ordering mistakes; if postcondition captures are older than the primary command's stdout/stderr/exit, classify the goal as `runner-error` unless direct sandbox state proves a product failure after the command completed
  - Use read-only commands in the sandbox when they materially improve confidence (for example: git log/status/show, ls/find/cat)
  - Do not speculate beyond the provided sandbox evidence and runner observations
  - For each failed goal, include a category:
    test-spec-error | tool-bug | runner-error | infrastructure-error | missing-artifact
  - For each goal, cite specific evidence (filenames, content snippets)
  - Follow the output format exactly
PROMPT

Instance Method Summary collapse

Instance Method Details

#prepare_runner(scenario:, sandbox_path:, test_cases: nil) ⇒ Hash

Parameters:

  • scenario (Models::TestScenario)
  • sandbox_path (String)
  • test_cases (Array<String>, nil) (defaults to: nil)

Returns:

  • (Hash) 


56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/ace/test/end_to_end_runner/molecules/pipeline_prompt_bundler.rb', line 56

def prepare_runner(scenario:, sandbox_path:, test_cases: nil)
  cache_dir = ensure_cache_dir(sandbox_path)
  system_path = File.join(cache_dir, "runner-system.md")
  prompt_path = File.join(cache_dir, "runner-prompt.md")

  File.write(system_path, RUNNER_SYSTEM_PROMPT)

  bundled = bundle_markdown_file(File.join(scenario.dir_path, "runner.yml.md"), test_cases: test_cases)
  bundled = bundled.gsub("Workspace root: (current directory)", "Workspace root: #{File.expand_path(sandbox_path)}")
  File.write(prompt_path, bundled)

  {
    system_path: system_path,
    prompt_path: prompt_path,
    output_path: File.join(cache_dir, "runner-output.md")
  }
end

#prepare_verifier(scenario:, sandbox_path:, test_cases: nil, runner_observations: nil, artifact_contract: nil) ⇒ Hash

Parameters:

  • scenario (Models::TestScenario)
  • sandbox_path (String)
  • test_cases (Array<String>, nil) (defaults to: nil)

Returns:

  • (Hash) 


78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/ace/test/end_to_end_runner/molecules/pipeline_prompt_bundler.rb', line 78

def prepare_verifier(scenario:, sandbox_path:, test_cases: nil, runner_observations: nil, artifact_contract: nil)
  cache_dir = ensure_cache_dir(sandbox_path)
  system_path = File.join(cache_dir, "verifier-system.md")
  prompt_path = File.join(cache_dir, "verifier-prompt.md")

  File.write(system_path, VERIFIER_SYSTEM_PROMPT)

  project_context = build_project_context_section(scenario)
  sandbox_context = build_sandbox_context_section(sandbox_path)
  artifacts = build_artifact_section(sandbox_path)
  contract = build_artifact_contract_section(artifact_contract)
  observations = build_runner_observation_section(runner_observations)
  criteria = bundle_markdown_file(File.join(scenario.dir_path, "verifier.yml.md"), test_cases: test_cases)
  File.write(prompt_path, [project_context, sandbox_context, artifacts, contract, observations, criteria].join("\n\n---\n\n"))

  {
    system_path: system_path,
    prompt_path: prompt_path,
    output_path: File.join(cache_dir, "verifier-output.md")
  }
end