Class: Phronomy::GeneratorVerifier

Inherits:
Object
  • Object
show all
Defined in:
lib/phronomy/generator_verifier.rb

Overview

Implements the Generator-Verifier multi-agent coordination pattern (Anthropic blog, Pattern 1): a generator agent produces an answer while a verifier agent evaluates its quality.

All prompt construction and result parsing are provided by the caller, giving full control over the LLM dialogue. The generator and verifier agents are configurable, and the pipeline retries until confidence passes the threshold or max iterations are reached.

Examples:

Basic usage with custom prompt builders

pipeline = Phronomy::GeneratorVerifier.new(
  draft_agent:           MyDraftAgent,
  review_agent:          MyReviewAgent,
  draft_prompt_builder:  ->(input, feedback) { "Question: #{input}" },
  review_prompt_builder: ->(input, draft, citations) { "Review: #{draft}" }
)
result = pipeline.invoke("What is the refund policy?")
puts result.output      # the final answer string
puts result.trusted?    # true when confidence >= threshold

Custom result parsers

pipeline = Phronomy::GeneratorVerifier.new(
  ...,
  draft_result_parser:  ->(text) { my_parse_draft(text) },
  review_result_parser: ->(text) { my_parse_review(text) }
)

Raising on low confidence

pipeline = Phronomy::GeneratorVerifier.new(
  ...,
  raise_if_untrusted: true
)
begin
  result = pipeline.invoke("question")
rescue Phronomy::LowConfidenceError => e
  puts "Untrusted: #{e.result.confidence}"
end

See Also:

Defined Under Namespace

Classes: Result

Constant Summary collapse

DEFAULT_CONFIDENCE_THRESHOLD =

Default confidence threshold for trusting an answer.

0.7
DEFAULT_MAX_ITERATIONS =

Default maximum draft-review cycles before returning best effort.

3

Instance Method Summary collapse

Constructor Details

#initialize(draft_agent:, review_agent:, draft_prompt_builder:, review_prompt_builder:, draft_result_parser: nil, review_result_parser: nil, confidence_threshold: DEFAULT_CONFIDENCE_THRESHOLD, max_iterations: DEFAULT_MAX_ITERATIONS, raise_if_untrusted: false) ⇒ GeneratorVerifier

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of GeneratorVerifier.

Parameters:

  • draft_agent (Class) —

    subclass of Phronomy::Agent::Base used to generate answer drafts

  • review_agent (Class) —

    subclass of Phronomy::Agent::Base used to evaluate each draft

  • draft_prompt_builder (#call) —

    +call(input, feedback)+ → String prompt for the generator. +feedback+ is nil on the first iteration and contains the reviewer's feedback string on subsequent iterations.

  • review_prompt_builder (#call) —

    +call(input, draft, citations)+ → String prompt for the verifier. +citations+ is an Array of Hashes.

  • draft_result_parser (#call, nil) (defaults to: nil) —

    +call(text)+ → Hash with +:answer+, +:confidence+, and +:citations+ keys. Defaults to JSON parsing with a safe fallback when the response cannot be parsed.

  • review_result_parser (#call, nil) (defaults to: nil) —

    +call(text)+ → Hash with +:approved+, +:score+, and +:feedback+ keys. Defaults to JSON parsing with a safe fallback.

  • confidence_threshold (Float) (defaults to: DEFAULT_CONFIDENCE_THRESHOLD) —

    minimum combined confidence to trust an answer (default: 0.7)

  • max_iterations (Integer) (defaults to: DEFAULT_MAX_ITERATIONS) —

    maximum draft-review cycles before returning the best-effort answer (default: 3)

  • raise_if_untrusted (Boolean) (defaults to: false) —

    when +true+, raises LowConfidenceError if the final result does not meet the confidence threshold (default: false)



117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/phronomy/generator_verifier.rb', line 117

def initialize(
  draft_agent:,
  review_agent:,
  draft_prompt_builder:,
  review_prompt_builder:,
  draft_result_parser: nil,
  review_result_parser: nil,
  confidence_threshold: DEFAULT_CONFIDENCE_THRESHOLD,
  max_iterations: DEFAULT_MAX_ITERATIONS,
  raise_if_untrusted: false
)
  @draft_agent_class = draft_agent
  @review_agent_class = review_agent
  @draft_prompt_builder = draft_prompt_builder
  @review_prompt_builder = review_prompt_builder
  @draft_result_parser = draft_result_parser || method(:default_parse_draft)
  @review_result_parser = review_result_parser || method(:default_parse_review)
  @threshold = confidence_threshold.to_f
  @max_iterations = max_iterations.to_i
  @raise_if_untrusted = raise_if_untrusted
  @compiled_workflow = nil
end

Instance Method Details

#invoke(input, config: {}) ⇒ Result

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Run the generator-verifier pipeline.

Parameters:

  • input (String) —

    the user question or task description

  • config (Hash) (defaults to: {}) —

    forwarded to the underlying agents (e.g. thread_id)

Returns:

Raises:



148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
 
# File 'lib/phronomy/generator_verifier.rb', line 148

def invoke(input, config: {})
  app = compiled_workflow
  state = app.invoke({input: input}, config: config)
  confidence = combined_confidence(state)
  trusted = confidence >= @threshold
  result = Result.new(
    output: state.output || state.draft.to_s,
    confidence: confidence,
    citations: state.citations,
    iterations: state.iteration,
    review_notes: state.review_notes,
    trusted: trusted
  )
  raise LowConfidenceError.new(result) if @raise_if_untrusted && !trusted
  result
end