Class: Rubino::Memory::Backends::Sqlite

Inherits:
Rubino::Memory::Backend show all
Includes:
AuxRetry, SalienceGate, SqliteExtraction, SqliteGraph
Defined in:
lib/rubino/memory/backends/sqlite.rb

Overview

LLM-extracted, bi-temporal fact store on embedded SQLite with hybrid recall — minus a graph DB, a server, or a multi-LLM-call pipeline.

Three ideas drive the design:

* ATOMIC LLM-extracted facts (one declarative fact per row), via a
  single aux-LLM call per turn that both ADDs new facts and SUPERSEDES
  contradicted ones (Graphiti edge-invalidation, collapsed to 1 call).
* BI-TEMPORAL supersession — a contradicted fact is soft-retired
  (valid_to set), not deleted; "live" memory = valid_to IS NULL, so we
  get temporal correctness without losing provenance.
* HYBRID ranked recall — FTS5/BM25 (+ optional vector KNN) fused with
  Reciprocal Rank Fusion and lightly kind-weighted, top-k under the
  char budget. Graph (1-hop) and recency are tail SUPPLEMENTS that only
  backfill the budget after direct content matches — never outranking
  them. (Optional vector KNN via sqlite-vec when available; see #vector?.)

The injection-defense floor (ThreatScanner + char-budget) is enforced on the write path exactly as Memory::Store does, so no fact can splice tainted or over-budget content into a future system prompt.

Constant Summary collapse

TABLE = 
:memory_facts
FTS = 
:memory_facts_fts
RRF_K = 
60
DEFAULT_K = 
20
DEFAULT_EXTRACT_MAX_RETRIES =

Bounded retry budget for the aux extraction call on a transient error (429/overloaded/5xx). Small by design: extraction is best-effort background work, and the per-session cursor re-feeds an exhausted turn next time, so we ride out a brief rate-limit window without piling up background backoff. Overridable via ‘memory.extract_max_retries`.

3
FTS_WEIGHT =

Weighted-RRF list weights for the DIRECT relevance signals (FTS/BM25 and vector KNN). Graph (1-hop) and recency are no longer fused here — they are tail supplements (see #rank) so they can never outrank a direct content match.

3.0
VECTOR_WEIGHT = 
3.0
STOPWORDS =

Trivial words that appear in almost every fact (“user”, “project”) or carry no retrieval signal — excluded from the FTS MATCH so a probe like “what package manager does the user use” doesn’t match every “User …” fact on the word “user”.

%w[
  the a an of to in on at for and or is are was were be been being do does did
  how what where when which who whom whose why this that these those it its
  use uses used user users project projects right now
].to_set.freeze
USER_KIND =

Maps the backend’s fact ‘kind` onto Memory::Store’s budget group so a user_profile fact is metered against the user budget and everything else against the shared memory budget — same split as the default backend.

"user_profile"
KIND_WEIGHT =

Light kind weighting applied after RRF so durable user facts outrank one-off facts on ties.

Hash.new(1.0).merge(
  "user_profile" => 1.3,
  "preference" => 1.2,
  "env" => 1.1
).freeze

Constants included from SalienceGate

SalienceGate::DURABLE_SIGNALS, SalienceGate::MIN_CONTENT_WORDS, SalienceGate::TOOL_LIMITATION_CLAIM, SalienceGate::TRIVIAL_WORDS

Constants included from SqliteGraph

SqliteGraph::CO_OCCURS, SqliteGraph::EDGES, SqliteGraph::ENTITIES

Class Method Summary collapse

Instance Method Summary collapse

Methods included from AuxRetry

#with_aux_retry

Methods included from SalienceGate

informative_words, salient?, tool_limitation_claim?, user_lines

Methods included from SqliteExtraction

#advance_extraction_cursor, #live_facts_for_prompt, #parse_json, #turn_text, #unextracted_messages

Methods included from SqliteGraph

#facts_tagged_with, #graph_neighbors, #index_fact_graph, #normalize_entity_name, #resolve_entity, #seed_entities, #supersede_edge, #upsert_edge

Constructor Details

#initialize(config: nil, db: nil, aux_client: nil) ⇒ Sqlite

Returns a new instance of Sqlite.



82
83
84
85
86
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 82

def initialize(config: nil, db: nil, aux_client: nil)
  super(config: config)
  @db = db || Rubino.database.db
  @aux_client = aux_client
end

Class Method Details

.backend_name ⇒ Object 



78
79
80
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 78

def self.backend_name
  "sqlite"
end

Instance Method Details

#available? ⇒ Boolean

FTS5 ships with the sqlite3 gem, so the backend is always available. (Vector mode is a best-effort upgrade gated separately by #vector?.)

Returns:

  • (Boolean) 


90
91
92
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 90

def available?
  true
end

#count ⇒ Object

Count only LIVE facts (valid_to IS NULL) — retired/superseded rows are tombstones the admin surface and #list already hide.



252
253
254
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 252

def count
  live_dataset.count
end

#delete(id) ⇒ Object 



239
240
241
242
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 239

def delete(id)
  row = resolve_row(id)
  row ? @db[TABLE].where(id: row[:id]).delete.positive? : false
end

#extract(session_id) ⇒ Object

ONE aux-LLM call over the turn’s NEW messages: returns supersede. Apply is pure Ruby — insert adds (deduped + guarded), retire superseded rows and insert their replacement.

Per-session cursor (#249): only messages newer than the session’s ‘memory_extracted_msg_id` watermark are fed, so each turn’s extraction is bounded to that turn’s new messages instead of an overlapping recency window. When a turn added nothing new past the cursor, we skip the aux-LLM call entirely (no redundant duplicate extraction pass), and advance the cursor only once the apply has landed.



153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 153

def extract(session_id)
  new_messages = unextracted_messages(session_id)
  turn = turn_text(new_messages)
  return [] if turn.strip.empty?

  # Salience gate (r5 F5/F6/F7): a greeting, a one-word "help", or any
  # turn whose USER text asserts nothing durable is a NOOP — skip the aux
  # call AND advance the cursor so it never mints a fact nor gets re-fed.
  unless salient?(turn)
    advance_extraction_cursor(session_id, new_messages)
    return []
  end

  result = call_llm(session_id: session_id, turn: turn)
  # A nil result means the aux call failed/parsed to nothing — leave the
  # cursor put so this turn's messages are retried next time rather than
  # silently dropped. A parsed result (even an empty {add,supersede})
  # means these messages WERE processed: advance the watermark so they're
  # never re-fed, which is the overlapping-window re-work #249 removes.
  return [] unless result

  stored = apply(result, session_id)
  advance_extraction_cursor(session_id, new_messages)
  stored
end

#find(id) ⇒ Object 



234
235
236
237
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 234

def find(id)
  row = resolve_row(id)
  row && present(row)
end

#forget(kind:, old_text:) ⇒ Object

Hard-delete the first LIVE fact of ‘kind` whose text includes `old_text` (forget = remove from the record entirely, vs supersede).



134
135
136
137
138
139
140
141
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 134

def forget(kind:, old_text:)
  target = live_dataset.where(kind: normalize_kind(kind))
                       .where(Sequel.like(:text, "%#{old_text}%")).first
  return nil unless target

  @db[TABLE].where(id: target[:id]).delete
  target
end

#list(kind: nil, limit: 20, include_retired: false) ⇒ Object

LIVE facts only by default — a superseded fact is a tombstone, not a current memory, so listing it undecorated next to its replacement presents contradicted data as true and makes the rows disagree with #count/#retrieve (#82). ‘include_retired: true` opts into the full supersession history (`rubino memory list –all`).



228
229
230
231
232
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 228

def list(kind: nil, limit: 20, include_retired: false)
  ds = (include_retired ? @db[TABLE] : live_dataset).order(Sequel.desc(:created_at)).limit(limit)
  ds = ds.where(kind: normalize_kind(kind)) if kind
  ds.all.map { |r| present(r) }
end

#project_context ⇒ Object 



192
193
194
195
196
197
198
199
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 192

def project_context
  return nil unless @config.dig("memory", "project_context_enabled")

  rows = live_dataset.where(kind: %w[project env]).order(Sequel.desc(:created_at)).limit(10).all
  return nil if rows.empty?

  rows.map { |r| r[:text] }.join("\n")
end

#replace(kind:, old_text:, content:) ⇒ Object

Replace the first LIVE fact of ‘kind` whose text includes `old_text`. Modelled as a supersession so history is preserved.



117
118
119
120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 117

def replace(kind:, old_text:, content:)
  target = live_dataset.where(kind: normalize_kind(kind))
                       .where(Sequel.like(:text, "%#{old_text}%")).first
  return nil unless target

  # Retire first so the old row's chars free up before the new fact is
  # budget-checked (a same-size replace must always fit).
  new_id = SecureRandom.uuid
  retire!(target[:id], new_id)
  insert_fact(text: content, kind: target[:kind],
              entities: parse_entities(target[:entities_json]),
              source_session_id: target[:source_session_id], id: new_id)
  target
end

#resolve_row(id) ⇒ Object

Resolve a caller-supplied id to AT MOST ONE row (shared with Store, parameterized by this backend’s dataset).



246
247
248
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 246

def resolve_row(id)
  Memory.resolve_row(@db[TABLE], id)
end

#retrieve(session_id:, query: nil, k: DEFAULT_K) ⇒ Object

HYBRID recall over LIVE facts: FTS5/BM25 on ‘query` (and vector KNN when available) fused via RRF and kind-weighted as the direct relevance ranking, then graph/recency-supplemented and greedily packed under the memory char budget. Returns rows shaped like the default backend (kind:, content:, …) so the prompt assembler is unchanged.



206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 206

def retrieve(session_id:, query: nil, k: DEFAULT_K)
  ranked = rank(query: query, k: k)
  budget = @config.dig("memory", "memory_char_limit")
  selected = []
  total = 0
  ranked.each do |row|
    len = row[:text].to_s.length
    break if budget&.positive? && total + len > budget

    selected << present(row)
    total += len
  end
  selected
end

#store(kind:, content:, source_session_id: nil, confidence: 1.0, metadata: {}) ⇒ Object

– WRITE path –



96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 96

def store(kind:, content:, source_session_id: nil, confidence: 1.0, metadata: {})
  k = normalize_kind(kind)
  # Exact/normalized-verbatim dedup at the direct write seam (#Y4):
  # MemoryTool#add bypasses the extraction near-dup gate, so the same fact
  # saved twice used to mint two identical rows. Idempotent — a verbatim
  # repeat (or whitespace/case variant) returns the existing row.
  existing = verbatim_duplicate(k, content)
  return present(existing) if existing

  insert_fact(
    text: content,
    kind: k,
    entities: Array(metadata[:entities]),
    source_session_id: source_session_id,
    confidence: confidence,
    valid_from: metadata[:valid_from]
  )
end

#user_profile ⇒ Object

– READ path –



181
182
183
184
185
186
187
188
189
190
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 181

def 
  return nil unless @config.dig("memory", "user_profile_enabled")

  rows = live_dataset.where(kind: USER_KIND).order(Sequel.desc(:created_at)).all
  return nil if rows.empty?

  text = rows.map { |r| r[:text] }.join("\n")
  limit = @config.dig("memory", "user_char_limit")
  text.length > limit ? text[0...limit] : text
end