Class: OKF::Bundle::Search

Inherits:
Object
  • Object
show all
Defined in:
lib/okf/bundle/search.rb,
lib/okf/bundle/search/scan.rb,
lib/okf/bundle/search/index.rb

Overview

Ranked text retrieval over one or more in-memory bundles. Terms are ANDed: every term must hit at least one searched field, though not necessarily the same one. Rows carry the fields each term hit, so a result stays explainable rather than being a bare relevance number.

This class is a facade. It owns everything that defines what a result is — the documents, the row and its key order, the snippet window, the final sort — and delegates only "which documents match, how well, and where" to an engine (Search::Index by default, Search::Scan for regexp). An engine that built its own rows could disagree about what a match is; this split makes that unrepresentable.

Pure — no disk, no stdio. The CLI's okf search and any embedding app share it: OKF::Bundle::Search.call(bundle, [ "dedup", "key" ]).

Defined Under Namespace

Modules: Index, Scan Classes: UnknownEngine, UnsupportedQuery

Constant Summary collapse

CAPABILITIES =

The declarable vocabulary: what an engine may claim about itself. Frozen so an engine declaring :regex is refused at registration rather than silently never selected — a typo in an addon would otherwise present as "my engine is ignored".

:prefix lives here and not in ROUTABLE on purpose. Nothing asks for its absence, so it selects nothing; what it does is document that this engine grows a term to the tokens it prefixes, which an FTS5 engine may not do by default. Declarative, and honest about being declarative.

%i[regexp fuzzy prefix].freeze
ROUTABLE =

The routable subset: the capabilities a query can actually require, and therefore the only ones that pick an engine. Kept distinct from CAPABILITIES because a capability nothing selects on, filed among the ones that do, is documentation posing as code.

Each entry is also the option name the facade hands an engine that declares it — see #engine_options, which is what keeps a meaningful option from reaching an engine that would quietly drop it.

%i[regexp fuzzy].freeze
DEFAULT_ENGINE =

Chosen when the query requires nothing in particular, which is the overwhelming majority of searches.

The scan, not the index, because a one-shot CLI builds an index, asks one question and exits — a build with a single query to amortize it over. Measured end to end: 3.00s vs 0.24s at 1,000 concepts, 0.83s vs 0.18s at 250, and the gap widens with the bundle. Raw-text matching also carries no tokenizer, so the terms that are glued to symbols and therefore unreachable by token (minifts, $OKF_HOME) stay findable by default.

What it gives up is BM25+ ranking, reachable with --engine index — and that is also the engine the browser page runs, so the two rank alike only when the index is named. See .okf/design/search-engines.md.

:scan
WEIGHTS =

The searchable fields with their rank weight, strongest signal first. In the index engine these ride as MiniFTS per-field boost; the scan sums the weights of the fields that matched instead.

{
  "title" => 5,
  "id" => 4,
  "tags" => 3,
  "type" => 2,
  "description" => 2,
  "body" => 1
}.freeze
FIELDS = 
WEIGHTS.keys.freeze
SNIPPET_FIELDS =

Fields whose match is only meaningful with surrounding context. The other fields already appear whole on the result row.

%w[description body].freeze
SNIPPET_RADIUS =

Characters of context kept on each side of the first matched term.

44
FUZZY_DISTANCE =

Edit distance as a fraction of term length, under fuzzy: true — the same 0.2 the browser page passes, so both forgive the same typos.

0.2
KEY_SEPARATOR =

The unique document key is "\0": ids are only unique within a bundle, and a merge that collided two bundles' same-named concepts would silently drop one.

"\0"

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil) ⇒ Search

Raises RegexpError on an invalid pattern with regexp: true, and UnsupportedQuery when no engine can answer — the caller owns turning either into a usage error. engines: overrides the registry, which is how the "nothing qualifies" path stays reachable without an addon installed.



193
194
195
196
197
198
199
200
201
202
 
# File 'lib/okf/bundle/search.rb', line 193

def initialize(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil)
  @bundles = bundles
  @terms = Array(terms).reject { |term| OKF.blank?(term) }.map(&:to_s)
  @fields = fields.nil? || fields.empty? ? FIELDS : fields
  @regexp = regexp
  @fuzzy = fuzzy
  @engine = engine
  @engines = engines
  @sources = {}
end

Class Method Details

.across(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil) ⇒ Object

Several bundles as [ slug, bundle ] pairs, ranked into one list with every row labeled by its slug. They share one index on purpose: BM25 weighs a term by how rare it is in the corpus, so per-bundle indexes would score the same match differently depending on which bundle it came from. One index makes one corpus, and the merged ranking is comparable by construction.



185
186
187
 
# File 'lib/okf/bundle/search.rb', line 185

def self.across(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil)
  new(bundles, terms, fields: fields, regexp: regexp, fuzzy: fuzzy, engine: engine, engines: engines).results
end

.call(bundle, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil) ⇒ Object 



176
177
178
 
# File 'lib/okf/bundle/search.rb', line 176

def self.call(bundle, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil)
  new([ [ nil, bundle ] ], terms, fields: fields, regexp: regexp, fuzzy: fuzzy, engine: engine, engines: engines).results
end

.engine_for(required, engines: self.engines, name: nil) ⇒ Object

The router. Naming an engine is an override, not a hint: it is how a caller reaches semantics no capability flag asks for — --engine scan means "match raw text", which the flags cannot express because there is nothing to require. A named engine that cannot do what was also asked is an error rather than a silent fallback, since falling back would answer a different question than the one that was posed.

Unnamed, the default engine leads, then registration order; the first available engine offering every required capability answers. Partition rather than sort_by, because sort_by is not stable and registration order is the tie-break.

Raises:



153
154
155
156
157
158
159
160
161
162
 
# File 'lib/okf/bundle/search.rb', line 153

def self.engine_for(required, engines: self.engines, name: nil)
  available = engines.select(&:available?)
  return named_engine(name, required, available) unless OKF.blank?(name)

  default, rest = available.partition { |engine| engine.id == DEFAULT_ENGINE }
  found = (default + rest).find { |engine| (required - engine.capabilities).empty? }
  return found if found

  raise UnsupportedQuery, required
end

.engines ⇒ Object

A frozen snapshot in registration order. Frozen because the registry is only meant to grow through .register, where the vocabulary is checked.



138
139
140
 
# File 'lib/okf/bundle/search.rb', line 138

def self.engines
  (@engines ||= []).dup.freeze
end

.register(engine) ⇒ Object

Append-only and idempotent by id: a second registration of an id already present is a no-op, so a double require cannot double the registry and an addon cannot quietly displace a built-in. Deliberately the same shape as the Linter's planned register hook — two extension points, one idiom.

Raises:

  • (ArgumentError) 


127
128
129
130
131
132
133
134
 
# File 'lib/okf/bundle/search.rb', line 127

def self.register(engine)
  rogue = engine.capabilities - CAPABILITIES
  raise ArgumentError, "unknown search capability: #{rogue.join(", ")}" unless rogue.empty?

  @engines ||= []
  @engines << engine unless @engines.any? { |registered| registered.id == engine.id }
  engine
end

Instance Method Details

#results ⇒ Object

Ranked match rows, catalog-style identity plus where the terms hit: [{ slug:, id:, title:, type:, area:, tags:, matched: [field, …], score:, snippet: }, …] ordered by score descending, then slug, then id. slug is present only when searching across bundles. No terms means no matches.



208
209
210
211
212
213
214
215
216
217
 
# File 'lib/okf/bundle/search.rb', line 208

def results
  return [] if @terms.empty?

  chosen = engine
  rows = chosen.call(documents, @terms, **engine_options(chosen)).map do |hit|
    slug, concept = @sources[hit[:key]]
    row(slug, concept, hit[:matched], hit[:score], hit[:terms])
  end
  rows.sort_by { |row| [ -row[:score], row[:slug].to_s, row[:id] ] }
end