Class: Woods::Evaluation::QuerySet

Inherits:
Object
  • Object
show all
Defined in:
lib/woods/evaluation/query_set.rb

Overview

Manages a set of evaluation queries with expected results.

Each query has a natural language question, a list of expected unit identifiers (ground truth), an intent classification, scope, and tags for filtering. QuerySets can be loaded from and saved to JSON files.

Examples:

qs = QuerySet.load("spec/fixtures/eval_queries.json")
qs.queries.each { |q| puts q.query }
qs.filter(intent: :lookup).size

Defined Under Namespace

Classes: Query

Constant Summary collapse

VALID_INTENTS = 
%i[lookup trace explain compare].freeze
VALID_SCOPES = 
%i[specific bounded broad].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(queries: []) ⇒ QuerySet

Initialize a QuerySet with an array of queries.

Parameters:

  • queries (Array<Query>) (defaults to: []) —

    Evaluation queries



40
41
42
 
# File 'lib/woods/evaluation/query_set.rb', line 40

def initialize(queries: [])
  @queries = queries
end

Instance Attribute Details

#queries ⇒ Array<Query> (readonly)

Returns The queries in this set.

Returns:

  • (Array<Query>) —

    The queries in this set



35
36
37
 
# File 'lib/woods/evaluation/query_set.rb', line 35

def queries
  @queries
end

Class Method Details

.load(path) ⇒ QuerySet

Load a QuerySet from a JSON file.

Parameters:

  • path (String) —

    Path to JSON file

Returns:

Raises:



49
50
51
52
53
54
55
56
57
 
# File 'lib/woods/evaluation/query_set.rb', line 49

def self.load(path)
  data = JSON.parse(File.read(path))
  queries = data.fetch('queries', []).map { |q| parse_query(q) }
  new(queries: queries)
rescue JSON::ParserError => e
  raise Woods::Error, "Invalid JSON in query set: #{e.message}"
rescue Errno::ENOENT => e
  raise Woods::Error, "Query set file not found: #{e.message}"
end

Instance Method Details

#add(query) ⇒ void

This method returns an undefined value.

Add a query to this set.

Parameters:

  • query (Query) —

    Query to add

Raises:

  • (ArgumentError) —

    if intent or scope is invalid



89
90
91
92
 
# File 'lib/woods/evaluation/query_set.rb', line 89

def add(query)
  validate_query!(query)
  @queries << query
end

#filter(intent: nil, scope: nil, tags: nil) ⇒ Array<Query>

Filter queries by intent, scope, or tags.

Parameters:

  • intent (Symbol, nil) (defaults to: nil) —

    Filter by intent

  • scope (Symbol, nil) (defaults to: nil) —

    Filter by scope

  • tags (Array<String>, nil) (defaults to: nil) —

    Filter by tags (any match)

Returns:

  • (Array<Query>) —

    Matching queries



76
77
78
79
80
81
82
 
# File 'lib/woods/evaluation/query_set.rb', line 76

def filter(intent: nil, scope: nil, tags: nil)
  result = queries
  result = result.select { |q| q.intent == intent } if intent
  result = result.select { |q| q.scope == scope } if scope
  result = result.select { |q| (q.tags & tags).any? } if tags
  result
end

#save(path) ⇒ void

This method returns an undefined value.

Save this QuerySet to a JSON file.

Parameters:

  • path (String) —

    Path to write JSON file



63
64
65
66
67
68
 
# File 'lib/woods/evaluation/query_set.rb', line 63

def save(path)
  data = {
    'queries' => queries.map { |q| serialize_query(q) }
  }
  File.write(path, JSON.pretty_generate(data))
end

#size ⇒ Integer

Number of queries in this set.

Returns:

  • (Integer) 


97
98
99
 
# File 'lib/woods/evaluation/query_set.rb', line 97

def size
  @queries.size
end