Module: PWN::AI::Agent::Curriculum

Defined in:: lib/pwn/ai/agent/curriculum.rb

Overview

PWN::AI::Agent::Curriculum is Tier 4/5 of the pwn-ai reinforcement loop — the SELF-PLAY layer that turns the agent from a passive experience-recorder into an active learner:

S1  .practice        — Mistake-driven auto-curriculum. Reads
                     Mistakes.top(unresolved), asks Reflect to
                     generate 3 minimal reproducer prompts per
                     signature, self-plays each under Loop.run,
                     and auto-mistakes_resolve when Reward.judge
                     says the practice run solved it. THE AGENT
                     PRACTISES ITS OWN WEAKNESSES OVERNIGHT.
S2  .counterfactual  — On a repeated in-turn failure, forks: branch
                     A continues with the correction_hint, branch
                     B asks an alt persona for a different tool.
                     Reward.judge picks the winner; (loser,
                     winner) → Reward.record_preference. Real
                     advantage estimation, not imagined rollouts.
S3  .critic          — Constitutional critic persona with TOOL
                     ACCESS (can shell/extro_verify the claim).
                     Runs BEFORE note_outcome; its verdict feeds
                     Reward.judge and its concrete flaw becomes a
                     preference pair when the agent self-corrects.
S4  .red_team_plan   — After plan_first, an adversarial persona
                     reviews the plan against THIS host's
                     Metrics/Mistakes/extro_drift and injects a
                     pre-emptive correction_hint on the step it
                     predicts will fail.
C3  .hindsight       — Hindsight Experience Replay. On failure,
                     asks the judge "what DID this trajectory
                     accomplish?", relabels the episode with the
                     achieved-goal as success:true. Free positive
                     samples from failures — first HER on real
                     tool traces.
W2  .train_and_gate  — export_finetune + export_dpo → local LoRA
                     (unsloth/axolotl if installed) → replay
                     Mistakes.top on vN vs vN+1 → promote iff
                     resolved(N+1) > resolved(N). Fully autonomous
                     weight-level self-improvement with a
                     regression gate.
W3  .calibrate       — Tracks plan_first predicted p(success) vs
                     actual outcome → Brier score in Metrics.

All entry points are cron-safe (never raise into the caller) and depth-guarded via Swarm's Thread.current so a curriculum run cannot recurse into itself.

Constant Summary collapse

CURRICULUM_DIR =

File.join(Dir.home, '.pwn', 'curriculum')

MODELS_FILE =

File.join(CURRICULUM_DIR, 'models.json')

CRITIC_NAME =

'pwn_critic'

RED_TEAM_NAME =

'pwn_red_team'

ALT_NAME =

'pwn_alt'

COOLDOWN_FAIL_NIGHTS = Priority-fix 1 — N-night cooldown window for thrashing signatures and stale "needs_human" tags. Signatures that score ~0 for COOLDOWN_FAIL_NIGHTS consecutive practice nights are parked so the curriculum stops burning cycles on them.

COOLDOWN_FILE =

File.join(CURRICULUM_DIR, 'cooldown.json')

TRAINER_CLI = W2 trainer discovery/execution is pure Ruby filesystem + argv spawn. Never shell-out via backticks or shell-string Open3 for probe/install checks. unsloth/axolotl presence is inferred from PATH CLIs and site-packages layout; any external process uses argv arrays only.

{
  unsloth: %w[unsloth],
  axolotl: %w[axolotl]
}.freeze

TRAINER_MODULE_HINTS =

{
  unsloth: %w[unsloth/__init__.py unsloth/models/__init__.py],
  axolotl: %w[axolotl/__init__.py axolotl/cli/train.py]
}.freeze

DPO_DIR_CONST =

File.join(Dir.home, '.pwn', 'finetune')

Class Method Summary collapse

.authors ⇒ Object

Author(s)

0day Inc.
.calibrate(opts = {}) ⇒ Object

Supported Method Parameters

PWN::AI::Agent::Curriculum.calibrate(predicted:, actual:, engine:).
.counterfactual(opts = {}) ⇒ Object
.critic(opts = {}) ⇒ Object

Supported Method Parameters

v = PWN::AI::Agent::Curriculum.critic( request: 'required - user request', final: 'required - candidate final answer', session_id: 'optional - for evidence lookup' ).
.help ⇒ Object
Display Usage for this Module.
.hindsight(opts = {}) ⇒ Object

Supported Method Parameters

PWN::AI::Agent::Curriculum.hindsight( request: 'required - the FAILED goal', final: 'required - the final produced anyway', session_id: 'required - trajectory to relabel' ).
.offline_judge(opts = {}) ⇒ Object
Priority-fix 3 — Offline ORM/PRM pass over recent sessions so local :failure_only introspect does not starve the reward corpus.
.practice(opts = {}) ⇒ Object
.preference_balance(opts = {}) ⇒ Object
P5 — W1 diversity report so monoculture is visible before DPO export.
.red_team_plan(opts = {}) ⇒ Object

Supported Method Parameters

hint = PWN::AI::Agent::Curriculum.red_team_plan( request: 'required - user goal', plan: 'required - numbered plan text from plan_first' ).
.train_and_gate(opts = {}) ⇒ Object

Supported Method Parameters

r = PWN::AI::Agent::Curriculum.train_and_gate( base_model: 'optional - ollama base tag (default PWN::Env[:ollama][:model])', trainer: 'optional - :unsloth | :axolotl | :auto (default :auto)', dry_run: 'optional - export + build eval set but do not train (default true)' ).

Class Method Details

.authors ⇒ `Object`

Author(s): 0day Inc. support@0dayinc.com



1447
1448
1449

# File 'lib/pwn/ai/agent/curriculum.rb', line 1447

public_class_method def self.authors
  "AUTHOR(S):\n  0day Inc. <support@0dayinc.com>\n"
end

.calibrate(opts = {}) ⇒ `Object`

Supported Method Parameters

PWN::AI::Agent::Curriculum.calibrate(predicted:, actual:, engine:)

# File 'lib/pwn/ai/agent/curriculum.rb', line 605

public_class_method def self.calibrate(opts = {})
  p = opts[:predicted].to_f.clamp(0.0, 1.0)
  a = opts[:actual].to_f.clamp(0.0, 1.0)
  brier = (p - a)**2
  Metrics.record_calibration(predicted: p, actual: a, brier: brier, engine: opts[:engine]) if defined?(Metrics) && Metrics.respond_to?(:record_calibration)
  { predicted: p, actual: a, brier: brier.round(4) }
end

.counterfactual(opts = {}) ⇒ `Object`

# File 'lib/pwn/ai/agent/curriculum.rb', line 351

public_class_method def self.counterfactual(opts = {})
  return nil unless enabled?(key: :counterfactual)
  return nil if in_curriculum?

  request = opts[:request].to_s
  ensure_persona(name: ALT_NAME, role: 'You are an alternative-approach generator for pwn-ai. Given a failing tool call, propose ONE concrete DIFFERENT tool + args that would achieve the same sub-goal on this host. Reply with the tool call only, no prose.')

  branch_a = opts[:hint].to_s.strip
  branch_a = "retry #{opts[:name]} with corrected args" if branch_a.empty?
  branch_b = with_curriculum_guard do
    ask_persona(name: ALT_NAME, request: "Goal: #{request[0, 300]}\nFailing: #{opts[:name]}(#{opts[:args].to_s[0, 200]}) → #{opts[:error].to_s[0, 200]}\nPropose ONE different tool+args.")
  end
  return nil if branch_b.to_s.strip.empty?

  sa_h = score_branch_detailed(request: request, branch: branch_a)
  sb_h = score_branch_detailed(request: request, branch: branch_b)
  sa = sa_h[:score]
  sb = sb_h[:score]
  if sb > sa
    winner = branch_b
    loser = branch_a
    tag = :b
    wmeta = sb_h
  else
    winner = branch_a
    loser = branch_b
    tag = :a
    wmeta = sa_h
  end
  real_hit = sa_h[:mode] == :real_dispatch || sb_h[:mode] == :real_dispatch
  shape = real_hit ? :real_dispatch : :imagined
  if defined?(Reward)
    Reward.record_preference(
      prompt: "#{request} | failing: #{opts[:name]} → #{opts[:error]}",
      rejected: loser.to_s[0, 2_000],
      chosen: winner.to_s[0, 2_000],
      source: :counterfactual,
      shape: shape,
      meta: {
        a_score: sa, b_score: sb,
        a_mode: sa_h[:mode], b_mode: sb_h[:mode],
        winner_trace: wmeta[:trace].to_s[0, 500]
      }
    )
  end
  log(event: :counterfactual, data: { branch: tag, a: sa, b: sb, tool: opts[:name].to_s, shape: shape })
  { branch: tag, content: winner, score: [sa, sb].max, a: sa, b: sb, shape: shape, a_mode: sa_h[:mode], b_mode: sb_h[:mode] }
rescue StandardError => e
  warn "[pwn-ai/curriculum] counterfactual swallowed: #{e.class}: #{e.message}"
  nil
end

.critic(opts = {}) ⇒ `Object`

Supported Method Parameters

v = PWN::AI::Agent::Curriculum.critic( request: 'required - user request', final: 'required - candidate final answer', session_id: 'optional - for evidence lookup' )

Returns { verdict: :pass|:flaw, flaw:, confidence: }. On :flaw the (final, flaw) pair is recorded as a preference so a future self-correction becomes DPO signal.

# File 'lib/pwn/ai/agent/curriculum.rb', line 418

public_class_method def self.critic(opts = {})
  return { verdict: :pass, source: :disabled } unless enabled?(key: :critic) || opts[:text_only]
  return { verdict: :pass, source: :recursion } if in_curriculum?

  # P24 — text_only: single Reflect shot, no tool-armed persona swarm.
  # Used when budget_exhaustion_hot? so critic cannot thrash the budget
  # that auto_introspect is trying to protect.
  return critic_text_only(request: opts[:request], final: opts[:final], session_id: opts[:session_id]) if opts[:text_only]

  ensure_persona(name: CRITIC_NAME, role: "You are pwn-ai's constitutional critic. Given a REQUEST and a candidate ANSWER, find ONE concrete, verifiable flaw (wrong fact, missing step, unsupported claim, broken command). You MAY call shell / extro_verify / pwn_eval to check. If none found reply exactly: PASS. Otherwise reply: FLAW: <one line>.")
  reply = with_curriculum_guard do
    ask_persona(name: CRITIC_NAME, request: "REQUEST:\n#{opts[:request].to_s[0, 800]}\n\nANSWER:\n#{opts[:final].to_s[0, 2_000]}")
  end
  if reply.to_s.strip.upcase.start_with?('PASS')
    log(event: :critic, data: { verdict: :pass })
    { verdict: :pass, confidence: 0.7 }
  else
    flaw = reply.to_s.sub(/\AFLAW:\s*/i, '').strip[0, 300]
    Mistakes.record(tool: 'assistant_answer', error: "critic: #{flaw}", args: opts[:final].to_s[0, 200], session_id: opts[:session_id], source: :model) if defined?(Mistakes)
    # P9 — DPO pair geometry: rejected=bad final, chosen=REVISED full
    # answer (not "CORRECTION: flaw" prose). Prefer persona rewrite.
    if defined?(Reward) && !flaw.to_s.empty?
      revised = revise_after_flaw(
        request: opts[:request],
        final: opts[:final],
        flaw: flaw,
        session_id: opts[:session_id]
      )
      Reward.record_preference(
        prompt: opts[:request].to_s[0, 1_000],
        rejected: opts[:final].to_s[0, 2_000],
        chosen: revised,
        source: :critic,
        shape: :revised_answer,
        meta: { flaw: flaw.to_s[0, 200] }
      )
    end
    log(event: :critic, data: { verdict: :flaw, flaw: flaw.to_s[0, 200] })
    { verdict: :flaw, flaw: flaw, confidence: 0.7 }
  end
rescue StandardError => e
  { verdict: :pass, error: e.message }
end

.help ⇒ `Object`

Display Usage for this Module

# File 'lib/pwn/ai/agent/curriculum.rb', line 1453

public_class_method def self.help
  puts <<~USAGE
    USAGE:
      # Tier 4 — self-play
      PWN::AI::Agent::Curriculum.practice(limit: 3)                     # S1 + P14 trajectory DPO pairs + P17 budget-first
      PWN::AI::Agent::Curriculum.offline_judge(since_hours: 24)         # P3 offline ORM/PRM fill
      PWN::AI::Agent::Curriculum.preference_balance                     # P5 W1 diversity report
      PWN::AI::Agent::Curriculum.counterfactual(request:, name:, args:, error:, hint:)  # S2 A/B → DPO pair
      PWN::AI::Agent::Curriculum.critic(request:, final:)               # S3 tool-armed constitutional critic
      PWN::AI::Agent::Curriculum.red_team_plan(request:, plan:)         # S4 telemetry-grounded plan review
      PWN::AI::Agent::Curriculum.hindsight(request:, final:, session_id:) # C3 HER soft-relabel

      # Tier 5 — close the weight loop
      PWN::AI::Agent::Curriculum.train_and_gate(dry_run: true)          # W2 export-ready; P11 gate v2 promote only with trainer+dry_run:false
      PWN::AI::Agent::Curriculum.calibrate(predicted: 0.8, actual: 1.0) # W3 Brier → Metrics[:calibration]

      Cron self-improvement (seeded by PWN::Cron.install_defaults):
        curriculum_practice_nightly  0 3 * * *   Curriculum.practice(limit: 3)
        curriculum_offline_judge     30 3 * * *  Curriculum.offline_judge(since_hours: 24, limit: 40)
        curriculum_train_weekly      0 4 * * 0   Curriculum.train_and_gate(dry_run: true)
        learning_consolidate_nightly 0 5 * * *   Learning.consolidate

      Config (PWN::Env[:ai][:agent]) — nil = auto (ON for remote engines, OFF for ollama):
        :critic         - Boolean/nil, run S3 before every note_outcome
        :red_team_plan  - Boolean/nil, run S4 after every plan_first
        :counterfactual - Boolean/nil, run S2 on REPEAT_THRESHOLD
        :hindsight      - Boolean, HER soft-relabel failures (default true)
        :reward_llm     - Boolean/nil, force ORM/PRM LLM teacher (nil=on for remote)

      #{self}.authors
  USAGE
end

.hindsight(opts = {}) ⇒ `Object`

Supported Method Parameters

PWN::AI::Agent::Curriculum.hindsight( request: 'required - the FAILED goal', final: 'required - the final produced anyway', session_id: 'required - trajectory to relabel' )

# File 'lib/pwn/ai/agent/curriculum.rb', line 500

public_class_method def self.hindsight(opts = {})
  return nil unless enabled?(key: :hindsight, default: true)
  return nil unless reflect_available?

  req = "The agent FAILED at: #{opts[:request].to_s[0, 300]}\nBut it produced: #{opts[:final].to_s[0, 800]}\n\nIn ≤12 words, what goal DID this trajectory accomplish? Reply with the goal only, or NOTHING if truly nothing."
  achieved = Reflect.on(request: req, suppress_pii_warning: true).to_s.strip
  return nil if achieved.empty? || achieved.upcase == 'NOTHING' || achieved.length > 200

  Learning.note_outcome(task: achieved, success: 'soft', details: "HER-relabelled from failed: #{opts[:request].to_s[0, 100]}", session_id: opts[:session_id], tags: %w[hindsight her soft], score: 0.7) if defined?(Learning)
  { original: opts[:request].to_s[0, 100], achieved: achieved }
rescue StandardError
  nil
end

.offline_judge(opts = {}) ⇒ `Object`

Priority-fix 3 — Offline ORM/PRM pass over recent sessions so local :failure_only introspect does not starve the reward corpus. Cron this nightly. Never raises.

Supported Method Parameters

r = PWN::AI::Agent::Curriculum.offline_judge( since_hours: 'optional - lookback window (default 24)', limit: 'optional - max sessions to score (default 40)', prm: 'optional - also run Process Reward Model (default true)', commit: 'optional - write scores into learning/sentinel (default true)' )

# File 'lib/pwn/ai/agent/curriculum.rb', line 219

public_class_method def self.offline_judge(opts = {})
  return { skipped: 'no Sessions' } unless defined?(PWN::Sessions)
  return { skipped: 'no Reward' } unless defined?(Reward)

  since_h = (opts[:since_hours] || 24).to_i
  limit   = (opts[:limit] || 40).to_i
  do_prm  = opts.key?(:prm) ? opts[:prm] : true
  commit  = opts.key?(:commit) ? opts[:commit] : true
  cutoff  = Time.now.utc - (since_h * 3_600)

  # PWN::Sessions.list is arity-0 (no kwargs). Cap/sort after the call.
  sids = if PWN::Sessions.respond_to?(:list)
           Array(PWN::Sessions.list)
             .sort_by { |s| s.is_a?(Hash) ? s[:mtime].to_s : '' }
             .reverse
             .first(limit * 2)
             .map { |s| s.is_a?(Hash) ? (s[:id] || s[:session_id] || s['id'] || s['session_id']) : s.to_s }
         else
           dir = (PWN::Sessions::SESSIONS_DIR if defined?(PWN::Sessions::SESSIONS_DIR)) || File.join(Dir.home, '.pwn', 'sessions')
           Dir[File.join(dir, '*.jsonl')].sort_by { |f| -File.mtime(f).to_i }.first(limit * 2).map { |f| File.basename(f, '.jsonl') }
         end

  scored = []
  sids.first(limit * 3).each do |sid|
    break if scored.length >= limit

    t = begin
      PWN::Sessions.load(session_id: sid)
    rescue StandardError
      []
    end
    next if t.nil? || t.empty?

    mtime = begin
      path = File.join(
        (PWN::Sessions::SESSIONS_DIR if defined?(PWN::Sessions::SESSIONS_DIR)) || File.join(Dir.home, '.pwn', 'sessions'),
        "#{sid}.jsonl"
      )
      File.exist?(path) ? File.mtime(path).utc : nil
    rescue StandardError
      nil
    end
    next if mtime && mtime < cutoff

    if commit && defined?(Learning)
      prior = Learning.outcomes(limit: 500).find do |o|
        o[:session_id].to_s == sid.to_s && o[:score] && Array(o[:tags]).include?('offline_judge')
      end
      next if prior
    end

    user = t.reverse.find { |e| e[:role].to_s == 'user' }
    final = t.reverse.find { |e| e[:role].to_s == 'assistant' && !e[:content].to_s.start_with?('PLAN:') }
    next unless user && final

    req = user[:content].to_s
    fin = final[:content].to_s
    next if req.strip.empty? || fin.strip.empty?

    v = Reward.judge(request: req, final: fin, session_id: sid, commit: commit)
    Reward.prm(request: req, session_id: sid) if do_prm
    # P7/W3 — offline path must also fill calibration so the controller
    # (force plan_first/critic at n≥8) actually becomes reachable under
    # :failure_only local introspect. Pull p(success)= out of any PLAN.
    if commit
      plan = t.find { |e| e[:role].to_s == 'assistant' && e[:content].to_s.start_with?('PLAN:') }
      pred = plan && plan[:content].to_s[/p\(success\)\s*=\s*([01](?:\.\d+)?)/i, 1]
      if pred
        eng = (PWN::Env.dig(:ai, :active) if defined?(PWN::Env))
        calibrate(predicted: pred.to_f, actual: v[:score].to_f, engine: eng)
      end
    end
    if commit && defined?(Learning)
      Learning.note_outcome(
        task: req[0, 120],
        success: v[:score].to_f >= 0.6,
        score: v[:score],
        details: "offline_judge #{v[:verdict]}(#{v[:score]}) #{v[:rationale]}",
        session_id: sid,
        tags: %w[offline_judge auto]
      )
    end
    scored << { session_id: sid, score: v[:score], verdict: v[:verdict] }
  end

  mean = scored.empty? ? nil : (scored.sum { |r| r[:score].to_f } / scored.length).round(3)
  # P10 — keep R3 window warm so proxy_distrust can engage on local hosts
  warm = (Reward.warm_sentinel(limit: 120) if commit && defined?(Reward) && Reward.respond_to?(:warm_sentinel))
  out = {
    scored: scored.length, mean: mean, since_hours: since_h,
    results: scored.first(10), sentinel_warm: warm
  }
  log(event: :offline_judge, data: out)
  out
rescue StandardError => e
  { error: "#{e.class}: #{e.message}" }
end

.practice(opts = {}) ⇒ `Object`

# File 'lib/pwn/ai/agent/curriculum.rb', line 81

public_class_method def self.practice(opts = {})
  limit   = (opts[:limit] || 3).to_i
  per     = (opts[:prompts_per] || 2).to_i
  dry_run = opts[:dry_run] ? true : false
  return { skipped: 'recursion guard' } if in_curriculum?

  FileUtils.mkdir_p(CURRICULUM_DIR)
  # 2.4 / 2.5 / P1 — skip reward_signal + needs_code_change / parked
  # + cooldown thrash + needs_human. Over-fetch so filters still fill.
  fetch_n = [limit * 4, 12].max
  candidates = if defined?(Mistakes)
                 Mistakes.top(limit: fetch_n, unresolved_only: true, practiceable_only: true)
               else
                 []
               end
  cool = load_cooldown
  # P17 — prefer budget-exhaustion fingerprints (agent_loop / critic)
  # so nightly self-play attacks the #1 live skill gap first.
  candidates = candidates.sort_by do |m|
    t = m[:tool].to_s
    e = m[:error].to_s.downcase
    budget = t == 'agent_loop' || t == 'assistant_answer' ||
             e.include?('budget exhausted') || e.include?('iteration budget')
    [budget ? 0 : 1, -m[:count].to_i]
  end
  targets = candidates.reject { |m| practice_skip?(mistake: m, cooldown: cool) }.first(limit)
  results = []

  with_curriculum_guard do
    targets.each do |m|
      next if practice_skip?(mistake: m, cooldown: cool)

      prompts = generate_reproducers(mistake: m, count: [per, 2].max)
      runs = dry_run ? [] : prompts.map { |p| self_play(prompt: p, tag: "practice:#{m[:signature]}") }
      solved = runs.select { |r| r[:score].to_f >= 0.7 }
      mean = runs.empty? ? 0.0 : (runs.sum { |r| r[:score].to_f } / runs.length)
      resolved = false
      # 2.4 — auto-resolve only with N≥2 holdout successes + store trace
      # P23 — auto-resolve only with N≥2 holdouts at judge≥0.7 AND a
      # real tool trace (not empty-final luck). Budget fingerprints
      # additionally require mean holdout ≥0.7 and short-horizon tags.
      if solved.length >= 2 && defined?(Mistakes)
        best = solved.max_by { |r| r[:score] }
        winning = best[:trace].to_s.strip
        winning = best[:final].to_s.strip if winning.length < 20
        budgetish = %w[agent_loop assistant_answer].include?(m[:tool].to_s) ||
                    m[:error].to_s.downcase.include?('budget')
        # refuse resolve on budget targets when winning_trace is prose-only
        trace_ok = winning.length >= 20 && (
          !budgetish || winning.match?(/→|shell|pwn_eval|tool/i) || best[:final].to_s.length.between?(1, 800)
        )
        unless trace_ok
          bump_cooldown!(cooldown: cool, signature: m[:signature], mean: mean) unless dry_run
          results << {
            signature: m[:signature], tool: m[:tool], prompts: prompts,
            runs: runs.map { |r| { score: r[:score], verdict: r[:verdict] } },
            resolved: false, mean_score: mean.round(3),
            reason: 'holdouts_ok_but_trace_weak'
          }
          next
        end
        fix = best[:final].to_s.lines.first(3).join.strip[0, 400]
        Mistakes.resolve(
          signature: m[:signature],
          fix: "auto-curriculum: #{fix}",
          structured: {
            strategy: budgetish ? 'short_horizon_finish' : 'curriculum_practice',
            tool: m[:tool],
            holdout_tests: solved.map { |r| r[:prompt] || r[:request] }.compact.first(5),
            winning_trace: winning[0, 2_000]
          }
        )
        # P14 — trajectory-shaped curriculum pair (not first-3-lines fix prose).
        # Mistakes.resolve also lands a mistakes_resolve pair via winning_trace;
        # this :curriculum row keeps W1 source diversity honest.
        if defined?(Reward)
          rejected = m[:snippet].to_s
          rejected = "FAILING: tool=#{m[:tool]} err=#{m[:error]}" if rejected.strip.empty?
          chosen = if best[:trace].to_s.strip.length >= 20
                     parts = []
                     parts << "STRATEGY: curriculum_practice | #{m[:tool]}"
                     parts << "WINNING_TRACE:\n#{best[:trace].to_s[0, 3_000]}"
                     parts << "FINAL:\n#{best[:final].to_s[0, 800]}" unless best[:final].to_s.strip.empty?
                     parts.join("\n")
                   else
                     best[:final].to_s[0, 3_500]
                   end
          Reward.record_preference(
            prompt: (best[:prompt] || prompts.first).to_s,
            rejected: rejected[0, 2_000],
            chosen: chosen,
            source: :curriculum,
            shape: :winning_trace,
            meta: {
              signature: m[:signature],
              score: best[:score],
              holdouts: solved.length
            }
          )
        end
        resolved = true
        cool.delete(m[:signature].to_s)
      elsif !dry_run
        # P1 — track zero-progress nights; park after COOLDOWN_FAIL_NIGHTS
        bump_cooldown!(cooldown: cool, signature: m[:signature], mean: mean)
      end
      results << {
        signature: m[:signature], tool: m[:tool], prompts: prompts,
        runs: runs.map { |r| { score: r[:score], verdict: r[:verdict] } },
        resolved: resolved, mean_score: mean.round(3)
      }
    end
  end
  save_cooldown(cooldown: cool)
  log(event: :practice, data: results)
  {
    practiced: results.length,
    resolved: results.count { |r| r[:resolved] },
    skipped_cooldown: cool.count { |_, v| v[:fail_nights].to_i >= COOLDOWN_FAIL_NIGHTS },
    results: results,
    dry_run: dry_run
  }
rescue StandardError => e
  { error: "#{e.class}: #{e.message}" }
end

.preference_balance(opts = {}) ⇒ `Object`

P5 — W1 diversity report so monoculture is visible before DPO export.

# File 'lib/pwn/ai/agent/curriculum.rb', line 319

public_class_method def self.preference_balance(opts = {})
  return { total: 0 } unless defined?(Reward)

  # P15 — prefer Reward.preference_balance (geometry-aware + optional scrub).
  if Reward.respond_to?(:preference_balance)
    return Reward.preference_balance(
      limit: opts[:limit] || 10_000,
      scrub: opts.key?(:scrub) ? opts[:scrub] : false
    )
  end

  rows = Reward.preferences(limit: opts[:limit] || 10_000)
  by = Hash.new(0)
  rows.each { |r| by[r[:source].to_s] += 1 }
  total = rows.length
  frac = by.transform_values { |n| total.zero? ? 0.0 : (n.to_f / total).round(3) }
  monoculture = total.positive? && (by.values.max.to_f / total) > 0.7
  {
    total: total,
    by_source: by,
    fractions: frac,
    monoculture: monoculture,
    advice: if monoculture
              'W1 monoculture: enable :counterfactual/:critic or loosen S2 gates; DPO will overfit mistakes_resolve prose.'
            else
              'W1 source mix OK'
            end
  }
rescue StandardError => e
  { error: "#{e.class}: #{e.message}" }
end

.red_team_plan(opts = {}) ⇒ `Object`

Supported Method Parameters

hint = PWN::AI::Agent::Curriculum.red_team_plan( request: 'required - user goal', plan: 'required - numbered plan text from plan_first' )

# File 'lib/pwn/ai/agent/curriculum.rb', line 472

public_class_method def self.red_team_plan(opts = {})
  return nil unless enabled?(key: :red_team_plan)
  return nil if in_curriculum?

  ensure_persona(name: RED_TEAM_NAME, role: 'You are pwn-ai\'s adversarial plan reviewer. Given a numbered tool plan and telemetry from THIS host (tool success rates, known mistakes, environment drift), identify the ONE step most likely to fail and say why in ≤2 lines. Cite the metric/mistake/drift. If the plan is sound reply: SOUND.')
  telemetry = build_telemetry
  reply = with_curriculum_guard do
    ask_persona(name: RED_TEAM_NAME, request: "GOAL: #{opts[:request].to_s[0, 300]}\n\nPLAN:\n#{opts[:plan].to_s[0, 1_200]}\n\nHOST TELEMETRY:\n#{telemetry}")
  end
  return nil if reply.to_s.strip.upcase.start_with?('SOUND') || reply.to_s.strip.empty?

  "[pwn-ai/red_team] pre-emptive: #{reply.to_s.strip[0, 400]}"
rescue StandardError => e
  warn "[pwn-ai/curriculum] red_team_plan swallowed: #{e.class}: #{e.message}"
  nil
end

.train_and_gate(opts = {}) ⇒ `Object`

Supported Method Parameters

r = PWN::AI::Agent::Curriculum.train_and_gate( base_model: 'optional - ollama base tag (default PWN::Env[:ollama][:model])', trainer: 'optional - :unsloth | :axolotl | :auto (default :auto)', dry_run: 'optional - export + build eval set but do not train (default true)' )

Best-effort orchestrator. When a supported trainer is installed it produces ~/.pwn/finetune/pwn-vN/, cuts an ollama Modelfile with the LoRA adapter, then REPLAYS Mistakes.top against vN and vN+1 under Reward.judge. Promotes (writes MODELS_FILE) iff vN+1 resolves more signatures. When no trainer is present it still exports SFT+DPO and emits the exact CLI to run manually — so the pipeline is complete even on a box without GPU.

# File 'lib/pwn/ai/agent/curriculum.rb', line 533

public_class_method def self.train_and_gate(opts = {})
  dry_run = opts.key?(:dry_run) ? opts[:dry_run] : true
  FileUtils.mkdir_p(CURRICULUM_DIR)
  sft = defined?(Learning) ? Learning.export_finetune(format: :sharegpt) : nil
  dpo = defined?(Reward) ? Reward.export_dpo : nil
  evalset = build_eval_set

  state = load_models
  version = state[:current].to_i + 1
  base = opts[:base_model] || (PWN::Env.dig(:ai, :ollama, :model) if defined?(PWN::Env)) || 'llama3'
  trainer = detect_trainer(preference: opts[:trainer])

  result = {
    version: version, base: base, trainer: trainer,
    sft: sft, dpo: dpo, eval_prompts: evalset.length, dry_run: dry_run
  }

  if dry_run || trainer.nil?
    result[:export_only] = true
    result[:weight_loop] = :export_ready
    result[:advice] = if trainer.nil?
                        'No trainer found — weight loop is EXPORT-ONLY on this host. ' \
                          'Install unsloth or axolotl on a GPU box, then re-run with dry_run:false. ' \
                          "Datasets ready at #{sft&.[](:path)} + #{dpo&.[](:path)}."
                      else
                        'dry_run — datasets + eval set exported; pass dry_run:false to train+gate+promote.'
                      end
    result[:manual_cli] = manual_train_cli(base: base, sft: sft, dpo: dpo, version: version)
    # P5 — surface W1 monoculture beside the export so operators see it
    result[:preference_balance] = begin
      preference_balance
    rescue StandardError
      nil
    end
    log(event: :train_and_gate, data: result)
    return result
  end

  adapter = run_trainer(trainer: trainer, base: base, sft: sft, dpo: dpo, version: version)
  return result.merge(error: 'trainer produced no adapter') unless adapter

  candidate = ollama_create(base: base, adapter: adapter, version: version)
  baseline  = state[:tag] || base
  # P11 — gate v2: resolved delta + mean judge + frozen smoke set.
  gate = ab_gate_v2(baseline: baseline, candidate: candidate, evalset: evalset)
  # P19 — refuse promote when W1 diet is still prose/monoculture.
  # Export-only is correct until scrubbed pairs show trajectory diversity.
  diet = preference_diet_gate
  gate = gate.merge(preference_diet: diet)
  promoted = gate[:promote] == true && diet[:ok] == true
  gate[:promote] = promoted
  gate[:promote_blocked_by_diet] = true unless diet[:ok]
  if promoted
    state[:previous] = state[:tag]
    state[:tag] = candidate
    state[:current] = version
    state[:promoted_at] = Time.now.utc.iso8601
    state[:gate] = gate
    save_models(state: state)
  end
  result.merge(adapter: adapter, candidate: candidate, gate: gate, promoted: promoted, weight_loop: :closed)
rescue StandardError => e
  { error: "#{e.class}: #{e.message}" }
end

Module: PWN::AI::Agent::Curriculum

Overview

Constant Summary collapse

Class Method Summary collapse

Class Method Details

.authors ⇒ Object

.calibrate(opts = {}) ⇒ Object

.counterfactual(opts = {}) ⇒ Object

.critic(opts = {}) ⇒ Object

.help ⇒ Object

.hindsight(opts = {}) ⇒ Object

.offline_judge(opts = {}) ⇒ Object

.practice(opts = {}) ⇒ Object

.preference_balance(opts = {}) ⇒ Object

.red_team_plan(opts = {}) ⇒ Object

.train_and_gate(opts = {}) ⇒ Object