Class: Woods::Evaluation::BaselineRunner

Inherits:
Object
  • Object
show all
Defined in:
lib/woods/evaluation/baseline_runner.rb

Overview

Runs simple baseline strategies for comparison against the full retrieval pipeline.

Provides three baseline strategies:

  • ‘:grep` — substring match on unit identifiers

  • ‘:random` — random selection from available units

  • ‘:file_level` — returns identifiers matching file paths

Examples:

runner = BaselineRunner.new(metadata_store: store)
results = runner.run("User model", strategy: :grep, limit: 10)
results  # => ["User", "UserProfile", "UserSerializer"]

Constant Summary collapse

VALID_STRATEGIES = 
%i[grep random file_level].freeze

Instance Method Summary collapse

Constructor Details

#initialize(metadata_store:, seed: nil) ⇒ BaselineRunner

Returns a new instance of BaselineRunner.

Parameters:

  • metadata_store (Object) —

    Store that responds to #all_identifiers and #find_by_type

  • seed (Integer, nil) (defaults to: nil) —

    Optional RNG seed for the ‘:random` baseline. Seeding makes evaluation runs reproducible — essential for comparing the real retriever against the baseline on the same query set across two invocations. `nil` (default) keeps the historical behavior of drawing from system entropy.



27
28
29
30
31
32
33
34
35
 
# File 'lib/woods/evaluation/baseline_runner.rb', line 27

def initialize(metadata_store:, seed: nil)
  @metadata_store = metadata_store
  @random = seed.nil? ? Random.new : Random.new(seed)
  # Ruby's `Random` instance isn't documented thread-safe; multiple
  # evaluator threads sharing one runner would otherwise interleave
  # `sample` calls and drift from the seeded sequence. A plain Mutex
  # around the read path is enough — `sample` is the only caller.
  @random_mutex = Mutex.new
end

Instance Method Details

#run(query, strategy:, limit: 10) ⇒ Array<String>

Run a baseline strategy for a query.

Parameters:

  • query (String) —

    Natural language query

  • strategy (Symbol) —

    Baseline strategy (:grep, :random, :file_level)

  • limit (Integer) (defaults to: 10) —

    Maximum number of results

Returns:

  • (Array<String>) —

    Unit identifiers

Raises:

  • (ArgumentError) —

    if the strategy is invalid



44
45
46
47
48
49
50
 
# File 'lib/woods/evaluation/baseline_runner.rb', line 44

def run(query, strategy:, limit: 10)
  unless VALID_STRATEGIES.include?(strategy)
    raise ArgumentError, "Invalid strategy: #{strategy}. Must be one of #{VALID_STRATEGIES.join(', ')}"
  end

  send(:"run_#{strategy}", query, limit)
end