Class: OKF::MCP::MemoryBackend

Inherits:
Object
  • Object
show all
Defined in:
lib/okf/mcp/memory_backend.rb

Overview

The residency cache and the corpus holder — the long-lived holder's branch of the kernel's lifecycle asymmetry. One parsed bundle per registered root, re-read only when the on-disk fingerprint (the markdown file list plus the newest mtime) moves; bodies always come from this layer, live from disk and canonical, whichever engine answers search. There is deliberately no disk cache here — the process is the cache; disk-side state is okf-sqlite3's territory.

Constant Summary collapse

FILTER_KEYS = 
%i[type tag dir status].freeze
MAX_CORPORA =

How many prepared corpora to hold at once, most-recently-used first. It used to be unbounded, keyed on the queried subset — a long-lived --http process answering bundles: ["a","b"], then ["a","c"], then ["b","c","d"] retained a full index for each of up to 2^n subsets until the box swapped. Four covers the real pattern ("*" plus a couple of working sets) and bounds the worst case at four indexes.

4

Instance Method Summary collapse

Constructor Details

#initialize ⇒ MemoryBackend

Returns a new instance of MemoryBackend.



23
24
25
26
27
28
 
# File 'lib/okf/mcp/memory_backend.rb', line 23

def initialize
  @mutex = Mutex.new
  @cache = {}
  @corpus_mutex = Mutex.new
  @corpora = {}
end

Instance Method Details

#capabilities ⇒ Object 



30
31
32
 
# File 'lib/okf/mcp/memory_backend.rb', line 30

def capabilities
  { name: "memory", ranked: false }
end

#catalog(root, filters = {}) ⇒ Object 



60
61
62
 
# File 'lib/okf/mcp/memory_backend.rb', line 60

def catalog(root, filters = {})
  folder(root).catalog.select { |row| matches?(row, filters) }
end

#during_request ⇒ Object

One unit of work, for the fingerprint memo below. The freshness check belongs between requests: within one, a tool asks the engine for the catalog and then reads the unparseable count off the same folder, and each ask re-walked the whole tree — a glob plus a stat per markdown file, all of it inside the lock.



97
98
99
100
101
102
103
 
# File 'lib/okf/mcp/memory_backend.rb', line 97

def during_request
  previous = Thread.current.thread_variable_get(:okf_mcp_prints)
  Thread.current.thread_variable_set(:okf_mcp_prints, {})
  yield
ensure
  Thread.current.thread_variable_set(:okf_mcp_prints, previous)
end

#folder(root) ⇒ Object

The disk handle behind read_concept, index and dirs — those stay on the parsed bundle whichever engine answers search: bodies are read live from disk (canonical), and the directory index needs the authored index.md bodies no derived store carries.



109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/okf/mcp/memory_backend.rb', line 109

def folder(root)
  @mutex.synchronize do
    entry = @cache[root]
    print = print_for(root)
    unless entry && entry[:fingerprint] == print
      folder = Bundle::Folder.load(root)
      # The threaded HTTP transport can hit these lazy memos from two
      # threads at once; build them once here, under this lock.
      folder.bundle.concept_by_id(nil)
      folder.bundle.paths_by_id
      folder.bundle.directories
      entry = { folder: folder, fingerprint: print }
      @cache[root] = entry
    end
    entry[:folder]
  end
end

#refresh(root) ⇒ Object 



34
35
36
37
 
# File 'lib/okf/mcp/memory_backend.rb', line 34

def refresh(root)
  folder(root)
  nil
end

#retain(roots) ⇒ Object

Drop every parsed bundle the caller no longer serves. The residency was bounded only by the served set being fixed at boot, and the registry re-read ended that: an operator repointing entries (a checkout per branch, rotating CI worktrees, remove-then-add) left every root the registry had ever named keyed here, each holding a parsed bundle with its id maps memoized, until the box swapped and every connected host dropped at once. The sibling corpus cache has been capped since it was written; this one lost its bound without gaining one.

An LRU would be the obvious cap and the wrong one: the natural bound is the served set, and a cap below it would thrash — re-parsing on every call for any operator who registered more bundles than the number guessed.

The corpus cache is pruned by the same rule, because it pins the same parsed bundles — plus a prepared index over each. Its LRU is not a substitute: eviction there happens only on an index query, which a scan-only workload never sends, so a corpus over a repointed root stayed resident indefinitely and the memory this prune claims to release was still held.



84
85
86
87
88
89
90
 
# File 'lib/okf/mcp/memory_backend.rb', line 84

def retain(roots)
  kept = {}
  roots.each { |root| kept[root] = true }
  @mutex.synchronize { @cache.delete_if { |root, _| !kept.key?(root) } }
  @corpus_mutex.synchronize { @corpora.delete_if { |key, _| key.any? { |root| !kept.key?(root) } } }
  nil
end

#search_pairs(pairs, terms, fields: nil, regexp: false, fuzzy: false, engine: nil) ⇒ Object

Engine doctrine, the CLI's exactly: the scan answers by default — no tokenizer recall holes, milliseconds over a resident bundle — and the index (BM25+, page parity) is opt-in by name or implied by fuzzy. The kernel routes and refuses incompatible asks (regexp needs the scan, fuzzy needs the index); those refusals surface as tool errors.

Index queries go through a held corpus: one shared index over the whole served set, so federated BM25 scores are comparable by construction (Search.across's own argument). The corpus is built on the first index query, not at boot, and dropped when any member bundle's fingerprint moves — a held index outliving its set is a wrong answer, not a slow one.



50
51
52
53
54
55
56
57
58
 
# File 'lib/okf/mcp/memory_backend.rb', line 50

def search_pairs(pairs, terms, fields: nil, regexp: false, fuzzy: false, engine: nil)
  bundles = pairs.map { |slug, root| [ slug, folder(root).bundle ] }
  options = { fields: fields, regexp: regexp, fuzzy: fuzzy, engine: engine }
  if index_query?(engine, fuzzy: fuzzy)
    Bundle::Search.with(corpus_for(pairs), terms, **options)
  else
    Bundle::Search.across(bundles, terms, **options)
  end
end