Class: OKF::Bundle::Search

Inherits:
Object
  • Object
show all
Defined in:
lib/okf/bundle/search.rb,
lib/okf/bundle/search/scan.rb,
lib/okf/bundle/search/index.rb

Overview

Ranked text retrieval over one or more in-memory bundles. Terms are ANDed: every term must hit at least one searched field, though not necessarily the same one. Rows carry the fields each term hit, so a result stays explainable rather than being a bare relevance number.

This class is a facade. It owns everything that defines what a result is — the documents, the row and its key order, the snippet window, the final sort — and delegates only "which documents match, how well, and where" to an engine (Search::Index by default, Search::Scan for regexp). An engine that built its own rows could disagree about what a match is; this split makes that unrepresentable.

Pure — no disk, no stdio. The CLI's okf search and any embedding app share it: OKF::Bundle::Search.call(bundle, [ "dedup", "key" ]).

Defined Under Namespace

Modules: Index, Scan Classes: Corpus, UnknownEngine, UnsupportedQuery

Constant Summary collapse

CAPABILITIES =

The declarable vocabulary: what an engine may claim about itself. Frozen so an engine declaring :regex is refused at registration rather than silently never selected — a typo in an addon would otherwise present as "my engine is ignored".

:prefix lives here and not in ROUTABLE on purpose. Nothing asks for its absence, so it selects nothing; what it does is document that this engine grows a term to the tokens it prefixes, which an FTS5 engine may not do by default. Declarative, and honest about being declarative.

%i[regexp fuzzy prefix].freeze
ROUTABLE =

The routable subset: the capabilities a query can actually require, and therefore the only ones that pick an engine. Kept distinct from CAPABILITIES because a capability nothing selects on, filed among the ones that do, is documentation posing as code.

Each entry is also the option name the facade hands an engine that declares it — see #engine_options, which is what keeps a meaningful option from reaching an engine that would quietly drop it.

%i[regexp fuzzy].freeze
DEFAULT_ENGINE =

Chosen when the query requires nothing in particular, which is the overwhelming majority of searches.

The scan, not the index, because a one-shot CLI builds an index, asks one question and exits — a build with a single query to amortize it over. Measured end to end: 3.00s vs 0.24s at 1,000 concepts, 0.83s vs 0.18s at 250, and the gap widens with the bundle. Raw-text matching also carries no tokenizer, so the terms that are glued to symbols and therefore unreachable by token (minifts, $OKF_HOME) stay findable by default.

What it gives up is BM25+ ranking, reachable with --engine index — and that is also the engine the browser page runs, so the two rank alike only when the index is named. See .okf/design/search-engines.md.

:scan
WEIGHTS =

The searchable fields with their rank weight, strongest signal first. In the index engine these ride as MiniFTS per-field boost; the scan sums the weights of the fields that matched instead.

{
  "title" => 5,
  "id" => 4,
  "tags" => 3,
  "type" => 2,
  "description" => 2,
  # A regression fix rather than a feature: in v0.1 a citation's text
  # lived in the body and was searchable at weight 1. After a bundle
  # migrates it lives in frontmatter, so without this a migrated bundle
  # silently loses the hit entirely. What this restores is *recall* at
  # the same weight — not an identical total. A v0.1 concept keeps
  # matching `body` too, because the text really is body prose there, so
  # it scores one higher than its migrated twin; that divergence is
  # deliberate and pinned by cli_twins_test.rb ("migrating moves a
  # source-only hit's snippet from body text to source text"). Suppressing
  # it would mean lying about `--in body` on a bundle whose body does
  # contain the words.
  "sources" => 1,
  "body" => 1
}.freeze
FIELDS = 
WEIGHTS.keys.freeze
SNIPPET_FIELDS =

Fields whose match is only meaningful with surrounding context. The other fields already appear whole on the result row. sources is here because indexed-but-un-snippeted would degrade a consumer's evidence line to a bare id list: after migration the snippet moves from body text to source text, it does not vanish.

%w[description body sources].freeze
SNIPPET_RADIUS =

Characters of context kept on each side of the first matched term.

44
FUZZY_DISTANCE =

Edit distance as a fraction of term length, under fuzzy: true — the same 0.2 the browser page passes, so both forgive the same typos.

0.2
KEY_SEPARATOR =

The unique document key is "\0": ids are only unique within a bundle, and a merge that collided two bundles' same-named concepts would silently drop one.

"\0"

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil, corpus: nil) ⇒ Search

Raises RegexpError on an invalid pattern with regexp: true, and UnsupportedQuery when no engine can answer — the caller owns turning either into a usage error. engines: overrides the registry, which is how the "nothing qualifies" path stays reachable without an addon installed.



287
288
289
290
291
292
293
294
295
296
297
 
# File 'lib/okf/bundle/search.rb', line 287

def initialize(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil, corpus: nil)
  @bundles = bundles
  @corpus = corpus
  @terms = Array(terms).reject { |term| OKF.blank?(term) }.map(&:to_s)
  @fields = fields.nil? || fields.empty? ? FIELDS : fields
  @regexp = regexp
  @fuzzy = fuzzy
  @engine = engine
  @engines = engines
  @sources = {}
end

Class Method Details

.across(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil) ⇒ Object

Several bundles as [ slug, bundle ] pairs, ranked into one list with every row labeled by its slug. They share one index on purpose: BM25 weighs a term by how rare it is in the corpus, so per-bundle indexes would score the same match differently depending on which bundle it came from. One index makes one corpus, and the merged ranking is comparable by construction.



200
201
202
 
# File 'lib/okf/bundle/search.rb', line 200

def self.across(bundles, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil)
  new(bundles, terms, fields: fields, regexp: regexp, fuzzy: fuzzy, engine: engine, engines: engines).results
end

.call(bundle, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil) ⇒ Object 



191
192
193
 
# File 'lib/okf/bundle/search.rb', line 191

def self.call(bundle, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil)
  new([ [ nil, bundle ] ], terms, fields: fields, regexp: regexp, fuzzy: fuzzy, engine: engine, engines: engines).results
end

.engine_for(required, engines: self.engines, name: nil) ⇒ Object

The router. Naming an engine is an override, not a hint: it is how a caller reaches semantics no capability flag asks for — --engine scan means "match raw text", which the flags cannot express because there is nothing to require. A named engine that cannot do what was also asked is an error rather than a silent fallback, since falling back would answer a different question than the one that was posed.

Unnamed, the default engine leads, then registration order; the first available engine offering every required capability answers. Partition rather than sort_by, because sort_by is not stable and registration order is the tie-break.

Raises:



168
169
170
171
172
173
174
175
176
177
 
# File 'lib/okf/bundle/search.rb', line 168

def self.engine_for(required, engines: self.engines, name: nil)
  available = engines.select(&:available?)
  return named_engine(name, required, available) unless OKF.blank?(name)

  default, rest = available.partition { |engine| engine.id == DEFAULT_ENGINE }
  found = (default + rest).find { |engine| (required - engine.capabilities).empty? }
  return found if found

  raise UnsupportedQuery, required
end

.engines ⇒ Object

A frozen snapshot in registration order. Frozen because the registry is only meant to grow through .register, where the vocabulary is checked.



153
154
155
 
# File 'lib/okf/bundle/search.rb', line 153

def self.engines
  (@engines ||= []).dup.freeze
end

.field_texts(concept) ⇒ Object

The searchable text of one concept, by field. Here rather than on an instance because a Corpus builds documents with no query in hand.



206
207
208
209
210
211
212
213
214
215
216
217
218
 
# File 'lib/okf/bundle/search.rb', line 206

def self.field_texts(concept)
  {
    "id" => concept.id,
    "title" => concept.title.to_s,
    "type" => concept.type.to_s,
    "description" => concept.description.to_s,
    "tags" => Array(concept.tags).join(" "),
    # Titles and resources together: a source is findable by what it is
    # called and by where it lives, which is how the body list read.
    "sources" => concept.sources.flat_map { |source| [ source["title"], source["resource"] ] }.compact.join(" "),
    "body" => concept.body
  }
end

.prepare(bundles, engine: nil, engines: nil) ⇒ Object

Prepare a corpus for a long-lived caller. Hand the result back to .with for every query.

engine: builds that engine's index now rather than on the first query. Without it the corpus holds only the documents, and the expensive half — the index — is still built lazily, which puts the whole cost on whoever searches first. A server knows its engine at boot, so it can pay there.



268
269
270
271
272
273
274
 
# File 'lib/okf/bundle/search.rb', line 268

def self.prepare(bundles, engine: nil, engines: nil)
  corpus = Corpus.new(bundles)
  return corpus if OKF.blank?(engine)

  corpus.index_for(engine_for([], engines: engines || self.engines, name: engine))
  corpus
end

.register(engine) ⇒ Object

Append-only and idempotent by id: a second registration of an id already present is a no-op, so a double require cannot double the registry and an addon cannot quietly displace a built-in. Deliberately the same shape as the Linter's planned register hook — two extension points, one idiom.

Raises:

  • (ArgumentError) 


142
143
144
145
146
147
148
149
 
# File 'lib/okf/bundle/search.rb', line 142

def self.register(engine)
  rogue = engine.capabilities - CAPABILITIES
  raise ArgumentError, "unknown search capability: #{rogue.join(", ")}" unless rogue.empty?

  @engines ||= []
  @engines << engine unless @engines.any? { |registered| registered.id == engine.id }
  engine
end

.with(corpus, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil) ⇒ Object

Query a prepared corpus. Same rows as .across, without rebuilding what the corpus already holds.



278
279
280
281
 
# File 'lib/okf/bundle/search.rb', line 278

def self.with(corpus, terms, fields: nil, regexp: false, fuzzy: false, engine: nil, engines: nil)
  new(corpus.bundles, terms, fields: fields, regexp: regexp, fuzzy: fuzzy,
    engine: engine, engines: engines, corpus: corpus).results
end

Instance Method Details

#results ⇒ Object

Ranked match rows, catalog-style identity plus where the terms hit: [{ slug:, id:, title:, type:, dir:, top_dir:, tags:, matched: [field, …], score:, snippet: }, …] ordered by score descending, then slug, then id. slug is present only when searching across bundles. No terms means no matches.



303
304
305
306
307
308
309
310
311
312
 
# File 'lib/okf/bundle/search.rb', line 303

def results
  return [] if @terms.empty?

  chosen = engine
  rows = chosen.call(documents, @terms, **engine_options(chosen)).map do |hit|
    slug, concept = @sources[hit[:key]]
    row(slug, concept, hit[:matched], hit[:score], hit[:terms])
  end
  rows.sort_by { |row| [ -row[:score], row[:slug].to_s, row[:id] ] }
end