Class: Woods::Evaluation::BaselineRunner

Inherits:
Object
  • Object
show all
Defined in:
lib/woods/evaluation/baseline_runner.rb

Overview

Runs simple baseline strategies for comparison against the full retrieval pipeline.

Provides three baseline strategies:

  • :grep — substring match on unit identifiers
  • :random — random selection from available units
  • :file_level — returns identifiers matching file paths

Examples:

runner = BaselineRunner.new(metadata_store: store)
results = runner.run("User model", strategy: :grep, limit: 10)
results  # => ["User", "UserProfile", "UserSerializer"]

Constant Summary collapse

VALID_STRATEGIES = 
%i[grep random file_level].freeze

Instance Method Summary collapse

Constructor Details

#initialize(metadata_store:, seed: nil) ⇒ BaselineRunner

Returns a new instance of BaselineRunner.

Parameters:

  • metadata_store (Object) —

    Store that responds to #all_identifiers and #find_by_type

  • seed (Integer, nil) (defaults to: nil) —

    Optional RNG seed for the :random baseline. Seeding makes evaluation runs reproducible — essential for comparing the real retriever against the baseline on the same query set across two invocations. nil (default) keeps the historical behavior of drawing from system entropy.



27
28
29
30
31
32
33
34
35
 
# File 'lib/woods/evaluation/baseline_runner.rb', line 27

def initialize(metadata_store:, seed: nil)
  @metadata_store = metadata_store
  @random = seed.nil? ? Random.new : Random.new(seed)
  # Ruby's `Random` instance isn't documented thread-safe; multiple
  # evaluator threads sharing one runner would otherwise interleave
  # `sample` calls and drift from the seeded sequence. A plain Mutex
  # around the read path is enough — `sample` is the only caller.
  @random_mutex = Mutex.new
end

Instance Method Details

#run(query, strategy:, limit: 10) ⇒ Array<String>

Run a baseline strategy for a query.

Parameters:

  • query (String) —

    Natural language query

  • strategy (Symbol) —

    Baseline strategy (:grep, :random, :file_level)

  • limit (Integer) (defaults to: 10) —

    Maximum number of results

Returns:

  • (Array<String>) —

    Unit identifiers

Raises:

  • (ArgumentError) —

    if the strategy is invalid



44
45
46
47
48
49
50
 
# File 'lib/woods/evaluation/baseline_runner.rb', line 44

def run(query, strategy:, limit: 10)
  unless VALID_STRATEGIES.include?(strategy)
    raise ArgumentError, "Invalid strategy: #{strategy}. Must be one of #{VALID_STRATEGIES.join(', ')}"
  end

  send(:"run_#{strategy}", query, limit)
end