Class: Gem::Skill::Verifier

Inherits:
Object
  • Object
show all
Defined in:
lib/gem/skill/verifier.rb

Overview

Second-pass quality gate for a generated SKILL.md.

Generation synthesizes prose sources (README, changelog, examples) which are frequently wrong or stale about exact signatures. The verifier re-checks the generated skill against the gem’s ACTUAL source code — the only source of truth — and corrects mismatched method signatures, default argument values, visibility, return values, and behavioral claims.

Whether the skill actually changed is decided by a deterministic diff of the content before and after, not by trusting the model’s self-report, so callers can rely on #changed? for an exit code and a “fixed” flag.

Defined Under Namespace

Classes: Result

Constant Summary collapse

BEGIN_MARK = 
"===BEGIN SKILL==="
END_MARK = 
"===END SKILL==="
SYSTEM_INSTRUCTIONS = 
<<~SYSTEM
  You verify a generated Claude Code SKILL.md for a Ruby gem against the gem's
  ACTUAL SOURCE CODE. The source code is the only source of truth. READMEs,
  changelogs, and docstrings are frequently stale or wrong; when the SKILL.md
  disagrees with the source, the source always wins.

  Check every concrete claim against the source: method signatures, default
  argument values, keyword vs positional arguments, public/private/protected
  visibility, return values, constant and class/module names, default option
  values, and described runtime behavior (including what arguments a yielded
  block actually receives). Correct anything the source contradicts.

  Rules:
  - Do NOT invent APIs, methods, or options that are absent from the source.
  - Do NOT restructure, re-style, or "improve" content that is already correct.
    Preserve correct text verbatim so the diff stays minimal.
  - Only change what the source proves is wrong.
SYSTEM
PROMPT = 
<<~PROMPT
  Verify the SKILL.md below for "%<gem_name>s" v%<version>s against the gem's
  source code. Correct every claim the source contradicts.

  Output ONLY the full corrected SKILL.md in raw Markdown (even if you change
  nothing), wrapped exactly between these marker lines and with no other text:
  %<begin_mark>s
  <corrected SKILL.md here>
  %<end_mark>s

  ============================================================
  CURRENT SKILL.md
  ============================================================

  %<skill>s

  ============================================================
  GEM SOURCE CODE (ground truth)
  ============================================================

  %<source>s
PROMPT

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(gem_name, version, model: Generator::DEFAULT_MODEL) ⇒ Verifier

Returns a new instance of Verifier.



73
74
75
76
77
 
# File 'lib/gem/skill/verifier.rb', line 73

def initialize(gem_name, version, model: Generator::DEFAULT_MODEL)
  @gem_name = gem_name
  @version  = version
  @model    = model
end

Instance Attribute Details

#gem_name ⇒ Object (readonly)

Returns the value of attribute gem_name.



71
72
73
 
# File 'lib/gem/skill/verifier.rb', line 71

def gem_name
  @gem_name
end

#model ⇒ Object (readonly)

Returns the value of attribute model.



71
72
73
 
# File 'lib/gem/skill/verifier.rb', line 71

def model
  @model
end

#version ⇒ Object (readonly)

Returns the value of attribute version.



71
72
73
 
# File 'lib/gem/skill/verifier.rb', line 71

def version
  @version
end

Instance Method Details

#verify(skill_content) ⇒ Object

Verify skill_content against the gem source. Returns a Result.



80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/gem/skill/verifier.rb', line 80

def verify(skill_content)
  fetcher = Fetcher.new(gem_name, version)
  source  = fetcher.source_code
  if source.nil? || source.strip.empty?
    return Result.new(content: skill_content, changed: false, verifiable: false, model: model)
  end

  raw       = build_chat.ask(format_prompt(skill_content, source)).content.to_s
  # Re-apply frontmatter to both sides so the diff compares like-for-like and
  # the stored skill always keeps valid frontmatter, even if the model dropped it.
  original  = Frontmatter.build(gem_name, version, skill_content)
  corrected = Frontmatter.build(gem_name, version, extract_skill(raw, skill_content))
  changed   = normalize(corrected) != normalize(original)

  Result.new(content: (changed ? corrected : skill_content), changed: changed,
             verifiable: true, model: model)
rescue RubyLLM::Error => e
  raise Error, e.message
end