Class: Rubino::Memory::Backends::Sqlite

Inherits:
Rubino::Memory::Backend show all
Includes:
AuxRetry, SalienceGate, SqliteExtraction, SqliteGraph
Defined in:
lib/rubino/memory/backends/sqlite.rb

Overview

“Tiny-Zep” memory backend on embedded SQLite (Zep/Graphiti-inspired, minus the graph DB, the server, and the six-LLM-call pipeline).

Three ideas are kept from Zep:

* ATOMIC LLM-extracted facts (one declarative fact per row), via a
  single aux-LLM call per turn that both ADDs new facts and SUPERSEDES
  contradicted ones (Graphiti edge-invalidation, collapsed to 1 call).
* BI-TEMPORAL supersession — a contradicted fact is soft-retired
  (valid_to set), not deleted; "live" memory = valid_to IS NULL, so we
  get temporal correctness without losing provenance.
* HYBRID ranked recall — FTS5/BM25 (+ optional vector KNN) fused with
  Reciprocal Rank Fusion and lightly kind-weighted, top-k under the
  char budget. Graph (1-hop) and recency are tail SUPPLEMENTS that only
  backfill the budget after direct content matches — never outranking
  them. (Optional vector KNN via sqlite-vec when available; see #vector?.)

The injection-defense floor (ThreatScanner + char-budget) is enforced on the write path exactly as Memory::Store does, so no fact can splice tainted or over-budget content into a future system prompt.

Constant Summary collapse

TABLE = 
:memory_facts
FTS = 
:memory_facts_fts
RRF_K = 
60
DEFAULT_K = 
20
DEFAULT_EXTRACT_MAX_RETRIES =

Bounded retry budget for the aux extraction call on a transient error (429/overloaded/5xx). Small by design: extraction is best-effort background work, and the per-session cursor re-feeds an exhausted turn next time, so we ride out a brief rate-limit window without piling up background backoff. Overridable via ‘memory.extract_max_retries`.

3
FTS_WEIGHT =

Weighted-RRF list weights for the DIRECT relevance signals (FTS/BM25 and vector KNN). Graph (1-hop) and recency are no longer fused here — they are tail supplements (see #rank) so they can never outrank a direct content match.

3.0
VECTOR_WEIGHT = 
3.0
STOPWORDS =

Trivial words that appear in almost every fact (“user”, “project”) or carry no retrieval signal — excluded from the FTS MATCH so a probe like “what package manager does the user use” doesn’t match every “User …” fact on the word “user”.

%w[
  the a an of to in on at for and or is are was were be been being do does did
  how what where when which who whom whose why this that these those it its
  use uses used user users project projects right now
].to_set.freeze
USER_KIND =

Maps the backend’s fact ‘kind` onto Memory::Store’s budget group so a user_profile fact is metered against the user budget and everything else against the shared memory budget — same split as the default backend.

"user_profile"
KIND_WEIGHT =

Light kind weighting applied after RRF so durable user facts outrank one-off facts on ties.

Hash.new(1.0).merge(
  "user_profile" => 1.3,
  "preference" => 1.2,
  "env" => 1.1
).freeze

Constants included from SalienceGate

SalienceGate::DURABLE_SIGNALS, SalienceGate::MIN_CONTENT_WORDS, SalienceGate::TRIVIAL_WORDS

Constants included from SqliteGraph

SqliteGraph::CO_OCCURS, SqliteGraph::EDGES, SqliteGraph::ENTITIES

Class Method Summary collapse

Instance Method Summary collapse

Methods included from AuxRetry

#with_aux_retry

Methods included from SalienceGate

informative_words, salient?, user_lines

Methods included from SqliteExtraction

#advance_extraction_cursor, #live_facts_for_prompt, #parse_json, #turn_text, #unextracted_messages

Methods included from SqliteGraph

#facts_tagged_with, #graph_neighbors, #index_fact_graph, #normalize_entity_name, #resolve_entity, #seed_entities, #supersede_edge, #upsert_edge

Constructor Details

#initialize(config: nil, db: nil, aux_client: nil) ⇒ Sqlite

Returns a new instance of Sqlite.



82
83
84
85
86
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 82

def initialize(config: nil, db: nil, aux_client: nil)
  super(config: config)
  @db = db || Rubino.database.db
  @aux_client = aux_client
end

Class Method Details

.backend_name ⇒ Object 



78
79
80
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 78

def self.backend_name
  "sqlite"
end

Instance Method Details

#available? ⇒ Boolean

FTS5 ships with the sqlite3 gem, so the backend is always available. (Vector mode is a best-effort upgrade gated separately by #vector?.)

Returns:

  • (Boolean) 


90
91
92
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 90

def available?
  true
end

#count ⇒ Object

Count only LIVE facts (valid_to IS NULL) — retired/superseded rows are tombstones the admin surface and #list already hide.



253
254
255
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 253

def count
  live_dataset.count
end

#delete(id) ⇒ Object 



231
232
233
234
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 231

def delete(id)
  row = resolve_row(id)
  row ? @db[TABLE].where(id: row[:id]).delete.positive? : false
end

#extract(session_id) ⇒ Object

ONE aux-LLM call over the turn’s NEW messages: returns supersede. Apply is pure Ruby — insert adds (deduped + guarded), retire superseded rows and insert their replacement.

Per-session cursor (#249): only messages newer than the session’s ‘memory_extracted_msg_id` watermark are fed, so each turn’s extraction is bounded to that turn’s new messages instead of an overlapping recency window. When a turn added nothing new past the cursor, we skip the aux-LLM call entirely (no redundant duplicate extraction pass), and advance the cursor only once the apply has landed.



145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 145

def extract(session_id)
  new_messages = unextracted_messages(session_id)
  turn = turn_text(new_messages)
  return [] if turn.strip.empty?

  # Salience gate (r5 F5/F6/F7): a greeting, a one-word "help", or any
  # turn whose USER text asserts nothing durable is a NOOP — skip the aux
  # call AND advance the cursor so it never mints a fact nor gets re-fed.
  unless salient?(turn)
    advance_extraction_cursor(session_id, new_messages)
    return []
  end

  result = call_llm(session_id: session_id, turn: turn)
  # A nil result means the aux call failed/parsed to nothing — leave the
  # cursor put so this turn's messages are retried next time rather than
  # silently dropped. A parsed result (even an empty {add,supersede})
  # means these messages WERE processed: advance the watermark so they're
  # never re-fed, which is the overlapping-window re-work #249 removes.
  return [] unless result

  stored = apply(result, session_id)
  advance_extraction_cursor(session_id, new_messages)
  stored
end

#find(id) ⇒ Object 



226
227
228
229
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 226

def find(id)
  row = resolve_row(id)
  row && present(row)
end

#forget(kind:, old_text:) ⇒ Object

Hard-delete the first LIVE fact of ‘kind` whose text includes `old_text` (forget = remove from the record entirely, vs supersede).



126
127
128
129
130
131
132
133
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 126

def forget(kind:, old_text:)
  target = live_dataset.where(kind: normalize_kind(kind))
                       .where(Sequel.like(:text, "%#{old_text}%")).first
  return nil unless target

  @db[TABLE].where(id: target[:id]).delete
  target
end

#list(kind: nil, limit: 20, include_retired: false) ⇒ Object

LIVE facts only by default — a superseded fact is a tombstone, not a current memory, so listing it undecorated next to its replacement presents contradicted data as true and makes the rows disagree with #count/#retrieve (#82). ‘include_retired: true` opts into the full supersession history (`rubino memory list –all`).



220
221
222
223
224
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 220

def list(kind: nil, limit: 20, include_retired: false)
  ds = (include_retired ? @db[TABLE] : live_dataset).order(Sequel.desc(:created_at)).limit(limit)
  ds = ds.where(kind: normalize_kind(kind)) if kind
  ds.all.map { |r| present(r) }
end

#project_context ⇒ Object 



184
185
186
187
188
189
190
191
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 184

def project_context
  return nil unless @config.dig("memory", "project_context_enabled")

  rows = live_dataset.where(kind: %w[project env]).order(Sequel.desc(:created_at)).limit(10).all
  return nil if rows.empty?

  rows.map { |r| r[:text] }.join("\n")
end

#replace(kind:, old_text:, content:) ⇒ Object

Replace the first LIVE fact of ‘kind` whose text includes `old_text`. Modelled as a supersession so history is preserved.



109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 109

def replace(kind:, old_text:, content:)
  target = live_dataset.where(kind: normalize_kind(kind))
                       .where(Sequel.like(:text, "%#{old_text}%")).first
  return nil unless target

  # Retire first so the old row's chars free up before the new fact is
  # budget-checked (a same-size replace must always fit).
  new_id = SecureRandom.uuid
  retire!(target[:id], new_id)
  insert_fact(text: content, kind: target[:kind],
              entities: parse_entities(target[:entities_json]),
              source_session_id: target[:source_session_id], id: new_id)
  target
end

#resolve_row(id) ⇒ Object

Resolve a caller-supplied id to AT MOST ONE row. A blank id resolves to nothing — a bare-prefix LIKE on “” matched the ‘%` wildcard → EVERY row, so `memory delete “”` wiped the whole store and reported success (#416). EXACT id wins; else accept a prefix ONLY when unambiguous (one match), so a short id from `memory list` works but a 1-char prefix can’t mass-select. limit(2) distinguishes “one” from “many”.



242
243
244
245
246
247
248
249
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 242

def resolve_row(id)
  key = id.to_s
  return nil if key.strip.empty?
  return @db[TABLE].where(id: key).first if @db[TABLE].where(id: key).get(:id)

  matches = @db[TABLE].where(Sequel.like(:id, "#{key}%")).limit(2).all
  matches.size == 1 ? matches.first : nil
end

#retrieve(session_id:, query: nil, k: DEFAULT_K) ⇒ Object

HYBRID recall over LIVE facts: FTS5/BM25 on ‘query` (and vector KNN when available) fused via RRF and kind-weighted as the direct relevance ranking, then graph/recency-supplemented and greedily packed under the memory char budget. Returns rows shaped like the default backend (kind:, content:, …) so the prompt assembler is unchanged.



198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 198

def retrieve(session_id:, query: nil, k: DEFAULT_K)
  ranked = rank(query: query, k: k)
  budget = @config.memory_char_limit
  selected = []
  total = 0
  ranked.each do |row|
    len = row[:text].to_s.length
    break if budget&.positive? && total + len > budget

    selected << present(row)
    total += len
  end
  selected
end

#store(kind:, content:, source_session_id: nil, confidence: 1.0, metadata: {}) ⇒ Object

– WRITE path –



96
97
98
99
100
101
102
103
104
105
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 96

def store(kind:, content:, source_session_id: nil, confidence: 1.0, metadata: {})
  insert_fact(
    text: content,
    kind: normalize_kind(kind),
    entities: Array(metadata[:entities]),
    source_session_id: source_session_id,
    confidence: confidence,
    valid_from: metadata[:valid_from]
  )
end

#user_profile ⇒ Object

– READ path –



173
174
175
176
177
178
179
180
181
182
 
# File 'lib/rubino/memory/backends/sqlite.rb', line 173

def 
  return nil unless @config.dig("memory", "user_profile_enabled")

  rows = live_dataset.where(kind: USER_KIND).order(Sequel.desc(:created_at)).all
  return nil if rows.empty?

  text = rows.map { |r| r[:text] }.join("\n")
  limit = @config.memory_user_char_limit
  text.length > limit ? text[0...limit] : text
end