Class: Aidp::Prompts::FeedbackCollector

Inherits:
Object
  • Object
show all
Defined in:
lib/aidp/prompts/feedback_collector.rb

Overview

Collects and stores feedback about prompt effectiveness

Tracks:

  • Prompt template usage
  • Completion outcomes (success/failure)
  • Iteration counts
  • User reactions
  • Suggested improvements

The collected feedback can be used for:

  • AGD pattern: Generating improved prompts based on feedback
  • Analytics: Understanding which prompts work well
  • Evolution: Automatic prompt improvement over time

Per issue #402: Integrates with TemplateVersionManager to:

  • Record positive votes (count but don't alter templates)
  • Record negative votes (trigger AGD for new template variants)

Examples:

Record feedback

collector = FeedbackCollector.new(project_dir: Dir.pwd)
collector.record(
  template_id: "decision_engine/condition_detection",
  outcome: :success,
  iterations: 5,
  context: { task: "classify API error" }
)

Constant Summary collapse

VERSION_MANAGER_FAILURE_THRESHOLD =

Threshold for logging error about persistent version manager failures

3

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(project_dir: Dir.pwd, repository: nil, version_manager: nil) ⇒ FeedbackCollector

Returns a new instance of FeedbackCollector.



41
42
43
44
45
46
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 41

def initialize(project_dir: Dir.pwd, repository: nil, version_manager: nil)
  @project_dir = project_dir
  @repository = repository || Database::Repositories::PromptFeedbackRepository.new(project_dir: project_dir)
  @version_manager_instance = version_manager
  @version_manager_failure_count = 0
end

Instance Attribute Details

#project_dir ⇒ Object (readonly)

Returns the value of attribute project_dir.



36
37
38
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 36

def project_dir
  @project_dir
end

#repository ⇒ Object (readonly)

Returns the value of attribute repository.



36
37
38
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 36

def repository
  @repository
end

Instance Method Details

#any? ⇒ Boolean

Check if any feedback exists

Returns:

  • (Boolean) 


141
142
143
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 141

def any?
  @repository.any?
end

#clear ⇒ Object

Clear all feedback data



132
133
134
135
136
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 132

def clear
  result = @repository.clear
  Aidp.log_info("feedback_collector", "feedback_cleared", count: result[:count])
  result
end

#entries(template_id: nil, outcome: nil, limit: 100) ⇒ Array<Hash>

Get feedback entries for analysis

Parameters:

  • template_id (String, nil) (defaults to: nil) —

    Filter by template (nil for all)

  • outcome (Symbol, nil) (defaults to: nil) —

    Filter by outcome

  • limit (Integer) (defaults to: 100) —

    Maximum entries to return

Returns:

  • (Array<Hash>) —

    Feedback entries



115
116
117
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 115

def entries(template_id: nil, outcome: nil, limit: 100)
  @repository.list(template_id: template_id, outcome: outcome, limit: limit)
end

#record(template_id:, outcome:, iterations: nil, user_reaction: nil, suggestions: nil, context: {}, track_version: true) ⇒ Object

Record feedback for a prompt template

Parameters:

  • template_id (String) —

    Template identifier

  • outcome (Symbol) —

    :success, :failure, :abandoned, :timeout

  • iterations (Integer, nil) (defaults to: nil) —

    Number of iterations to completion

  • user_reaction (Symbol, nil) (defaults to: nil) —

    :positive, :negative, :neutral

  • suggestions (Array<String>, nil) (defaults to: nil) —

    Improvement suggestions

  • context (Hash) (defaults to: {}) —

    Additional context (task type, error type, etc.)

  • track_version (Boolean) (defaults to: true) —

    Whether to update version manager (default: true)



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 62

def record(template_id:, outcome:, iterations: nil, user_reaction: nil, suggestions: nil, context: {}, track_version: true)
  Aidp.log_debug("feedback_collector", "recording_feedback",
    template_id: template_id,
    outcome: outcome,
    iterations: iterations,
    user_reaction: user_reaction)

  result = @repository.record(
    template_id: template_id,
    outcome: outcome,
    iterations: iterations,
    user_reaction: user_reaction,
    suggestions: suggestions,
    context: context
  )

  if result[:success]
    Aidp.log_info("feedback_collector", "feedback_recorded",
      template_id: template_id,
      outcome: outcome)

    # Per issue #402: Update version manager based on user reaction
    if track_version && user_reaction
      update_version_feedback(
        template_id: template_id,
        user_reaction: user_reaction,
        suggestions: suggestions,
        context: context
      )
    end
  else
    Aidp.log_warn("feedback_collector", "feedback_record_failed",
      template_id: template_id,
      error: result[:error])
  end

  result
end

#summary(template_id:) ⇒ Hash

Get feedback summary for a template

Parameters:

  • template_id (String) —

    Template identifier

Returns:

  • (Hash) —

    Summary statistics



105
106
107
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 105

def summary(template_id:)
  @repository.summary(template_id: template_id)
end

#templates_needing_improvement(min_uses: 5, max_success_rate: 70.0) ⇒ Array<Hash>

Get templates that need improvement based on feedback

Parameters:

  • min_uses (Integer) (defaults to: 5) —

    Minimum uses to consider

  • max_success_rate (Float) (defaults to: 70.0) —

    Success rate threshold (0-100)

Returns:

  • (Array<Hash>) —

    Templates needing improvement with summaries



124
125
126
127
128
129
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 124

def templates_needing_improvement(min_uses: 5, max_success_rate: 70.0)
  @repository.templates_needing_improvement(
    min_uses: min_uses,
    max_success_rate: max_success_rate
  )
end

#version_manager ⇒ Object

Lazily initialize version manager



49
50
51
 
# File 'lib/aidp/prompts/feedback_collector.rb', line 49

def version_manager
  @version_manager_instance ||= TemplateVersionManager.new(project_dir: @project_dir)
end