Class: ActiveGenie::Scorer::JuryBench

Inherits:
BaseModule
  • Object
show all
Defined in:
lib/active_genie/scorer/jury_bench.rb

Overview

The JuryBench class provides a foundation for Scorer text content against specified criteria using AI-powered evaluation. It supports both single and multiple jury scenarios, with the ability to automatically recommend juries when none are specified.

The Scorer process evaluates text based on given criteria and returns detailed feedback including individual jury scores, reasoning, and a final aggregated score.

Examples:

JuryBench usage with a single jury

JuryBench.call("Sample text", "Evaluate grammar and clarity", ["Grammar Expert"])

Usage with automatic jury recommendation

JuryBench.call("Sample text", "Evaluate technical accuracy")

Constant Summary collapse

PROMPT = 
File.read(File.join(__dir__, 'jury_bench.prompt.md'))

Instance Method Summary collapse

Methods inherited from BaseModule

call, #config

Constructor Details

#initialize(text, criteria, juries = [], config: {}) ⇒ Hash

Returns The evaluation result containing the scores and reasoning @return [Number] :final_score The final score of the text based on the criteria and juries @return [String] :final_reasoning Detailed explanation of why the final score was reached.

Parameters:

  • text (String) —

    The text content to be evaluated

  • criteria (String) —

    The evaluation criteria or rubric to assess against

  • juries (Array<String>) (defaults to: []) —

    Optional list of specific juries. If empty, juries will be automatically recommended based on the content and criteria

  • config (Hash) (defaults to: {}) —

    Additional configuration config that modify the Scorer behavior



30
31
32
33
34
35
 
# File 'lib/active_genie/scorer/jury_bench.rb', line 30

def initialize(text, criteria, juries = [], config: {})
  @text = text
  @criteria = criteria
  @param_juries = Array(juries).compact.uniq
  super(config:)
end

Instance Method Details

#call ⇒ Object 



37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
 
# File 'lib/active_genie/scorer/jury_bench.rb', line 37

def call
  messages = [
    {  role: 'system', content: PROMPT },
    {  role: 'user', content: "Scorer criteria: #{@criteria}" },
    {  role: 'user', content: "Text to score: #{@text}" }
  ]

  provider_response = ::ActiveGenie::Providers::UnifiedProvider.function_calling(
    messages,
    build_function,
    config:
  )

  ActiveGenie::Result.new(
    data: provider_response['final_score'] || 0,
    reasoning: provider_response['final_reasoning'],
    metadata: provider_response
  )
end