Class: Ace::Test::EndToEndRunner::Atoms::ResultParser

Inherits:
Object
  • Object
show all
Defined in:
lib/ace/test/end_to_end_runner/atoms/result_parser.rb

Overview

Parses structured JSON results from LLM responses

Extracts JSON from LLM text output, handling various formatting patterns including fenced code blocks and raw JSON.

Defined Under Namespace

Classes: ParseError

Class Method Summary collapse

Class Method Details

.extract_json(text) ⇒ String?

Extract JSON from text, handling code fences and raw JSON

Parameters:

  • text (String) —

    Text potentially containing JSON

Returns:

  • (String, nil) —

    Extracted JSON string or nil



34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/ace/test/end_to_end_runner/atoms/result_parser.rb', line 34

def self.extract_json(text)
  return nil if text.nil? || text.to_s.strip.empty?

  stripped = text.to_s.strip

  # Try to find JSON in code fences first
  match = stripped.match(/```(?:json)?\s*\n(.*?)\n\s*```/m)
  return match[1].strip if match

  # Treat unfenced content as JSON only when the whole payload is a JSON object.
  return stripped if stripped.start_with?("{") && stripped.end_with?("}")

  nil
end

.parse(text) ⇒ Hash

Parse LLM response text into a structured result hash

Parameters:

  • text (String) —

    Raw LLM response text

Returns:

  • (Hash) —

    Parsed result with :test_id, :status, :test_cases, :summary

Raises:

  • (ParseError) —

    If no valid JSON found in response



19
20
21
22
23
24
25
26
27
28
 
# File 'lib/ace/test/end_to_end_runner/atoms/result_parser.rb', line 19

def self.parse(text)
  json_str = extract_json(text)
  raise ParseError, "No JSON found in LLM response" if json_str.nil?

  parsed = JSON.parse(json_str, symbolize_names: true)
  validate_result(parsed)
  normalize_result(parsed)
rescue JSON::ParserError => e
  raise ParseError, "Invalid JSON in LLM response: #{e.message}"
end

.parse_tc(text) ⇒ Hash

Parse TC-level LLM response into a structured result hash

Handles single-TC JSON format with tc_id field. Falls back to parse() if the response contains multi-TC format.

Parameters:

  • text (String) —

    Raw LLM response text

Returns:

  • (Hash) —

    Parsed result with single-entry :test_cases array

Raises:

  • (ParseError) —

    If no valid JSON found in response



115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
# File 'lib/ace/test/end_to_end_runner/atoms/result_parser.rb', line 115

def self.parse_tc(text)
  json_str = extract_json(text)
  raise ParseError, "No JSON found in LLM response" if json_str.nil?

  parsed = JSON.parse(json_str, symbolize_names: true)

  # If response has test_cases array, delegate to standard parse
  return parse(text) if parsed.key?(:test_cases)

  validate_tc_result(parsed)
  normalize_tc_result(parsed)
rescue JSON::ParserError => e
  raise ParseError, "Invalid JSON in LLM response: #{e.message}"
end