Class: SkillBench::Judge::Prompt

Inherits:
Object
  • Object
show all
Defined in:
lib/skill_bench/judge/prompt.rb

Overview

Builds structured prompts for the LLM judge.

Assembles task description, evaluation criteria, skill context, and agent output into a single prompt for blind scoring. Untrusted content (task, skill context, and agent output) is wrapped in per-run random sentinel fences and stripped of that sentinel, so embedded text cannot forge a boundary and inject instructions into the judge.

Constant Summary collapse

SENTINEL_BYTES =

Byte length of the per-run sentinel; SecureRandom.hex yields 2x hex chars.

16

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(task:, criteria:, skill_context:, agent_output:) ⇒ Prompt

Returns a new instance of Prompt.

Parameters:

  • task (String) —

    The task description.

  • criteria (SkillBench::Criteria) —

    The eval criteria.

  • skill_context (String, nil) —

    The skill context XML (nil for baseline runs).

  • agent_output (String) —

    The agent output.



33
34
35
36
37
38
39
 
# File 'lib/skill_bench/judge/prompt.rb', line 33

def initialize(task:, criteria:, skill_context:, agent_output:)
  @task = task
  @criteria = criteria
  @skill_context = skill_context
  @agent_output = agent_output
  @sentinel = SecureRandom.hex(SENTINEL_BYTES)
end

Class Method Details

.call(task:, criteria:, skill_context:, agent_output:) ⇒ Hash

Builds the judge prompt.

Parameters:

  • task (String) —

    The task description from task.md.

  • criteria (SkillBench::Criteria) —

    The eval criteria with dimensions.

  • skill_context (String, nil) —

    XML-wrapped skill context (nil for baseline runs).

  • agent_output (String) —

    Git diff and agent summary.

Returns:

  • (Hash) —

    Service response with prompt or error.



25
26
27
 
# File 'lib/skill_bench/judge/prompt.rb', line 25

def self.call(task:, criteria:, skill_context:, agent_output:)
  new(task:, criteria:, skill_context:, agent_output:).call
end

Instance Method Details

#call ⇒ Hash

Assembles and returns the judge prompt.

Returns:

  • (Hash) —

    Service response with prompt or error.



44
45
46
47
48
49
50
51
52
53
54
55
 
# File 'lib/skill_bench/judge/prompt.rb', line 44

def call
  return missing_task_result if task.nil? || task.strip.empty?
  return missing_criteria_result if criteria.nil?
  return missing_agent_output_result if agent_output.nil? || agent_output.to_s.strip.empty?
  return missing_skill_context_result unless valid_skill_context?

  prompt = assemble_prompt
  { success: true, response: { prompt: prompt } }
rescue StandardError => e
  SkillBench::ErrorLogger.log_error(e, 'Judge::Prompt Build Error')
  { success: false, response: { error: { message: e.message } } }
end