Class: SwarmSDK::V3::Memory::Store

Inherits:
Object
  • Object
show all
Defined in:
lib/swarm_sdk/v3/memory/store.rb

Overview

Orchestrator for the memory system

Coordinates all memory components: adapter, embedder, retriever, context builder, ingestion pipeline, compressor, and consolidator. This is the main entry point for Agent to interact with memory.

Examples:

store = Store.new(adapter: adapter, embedder: embedder)
store.load

# Retrieve relevant context for a query
messages = store.build_context(query: "auth", recent_turns: turns)

# Ingest a new turn
store.ingest_turn(text: "We use JWT...", turn_id: "turn_001")

store.save

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(adapter:, embedder:, retrieval_top_k: 15, semantic_weight: 0.5, keyword_weight: 0.5, chat: nil, associative_memory: false) ⇒ Store

Returns a new instance of Store.

Parameters:

  • adapter (Adapters::Base) —

    Storage adapter

  • embedder (Embedder) —

    Text embedder

  • retrieval_top_k (Integer) (defaults to: 15) —

    Cards to retrieve per query

  • semantic_weight (Float) (defaults to: 0.5) —

    Semantic search weight

  • keyword_weight (Float) (defaults to: 0.5) —

    Keyword search weight

  • chat (RubyLLM::Chat, nil) (defaults to: nil) —

    LLM chat for compression/ingestion

  • associative_memory (Boolean) (defaults to: false) —

    Whether to enable associative memory



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 40

def initialize(adapter:, embedder:, retrieval_top_k: 15, semantic_weight: 0.5, keyword_weight: 0.5, chat: nil,
  associative_memory: false)
  @adapter = adapter
  @embedder = embedder
  @chat = chat

  @retriever = Retriever.new(
    adapter: adapter,
    embedder: embedder,
    semantic_weight: semantic_weight,
    keyword_weight: keyword_weight,
  )

  @context_builder = ContextBuilder.new(
    retriever: @retriever,
    adapter: adapter,
    retrieval_top_k: retrieval_top_k,
    embedder: embedder,
    associative_memory: associative_memory,
  )

  @ingestion_pipeline = IngestionPipeline.new(
    adapter: adapter,
    embedder: embedder,
    chat: chat,
  )

  @exposure_tracker = ExposureTracker.new(adapter)

  @compressor = Compressor.new(
    adapter: adapter,
    chat: chat,
  )

  @consolidator = Consolidator.new(
    adapter: adapter,
    embedder: embedder,
  )
end

Instance Attribute Details

#adapter ⇒ Adapters::Base (readonly)

Returns Storage adapter.

Returns:



25
26
27
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 25

def adapter
  @adapter
end

#embedder ⇒ Embedder (readonly)

Returns Text embedder.

Returns:



28
29
30
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 28

def embedder
  @embedder
end

#retriever ⇒ Retriever (readonly)

Returns Hybrid search retriever.

Returns:



31
32
33
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 31

def retriever
  @retriever
end

Instance Method Details

#build_context(query:, recent_turns: [], system_prompt: nil, read_only: false) ⇒ Array<Hash>

Build working context for a query

Parameters:

  • query (String) —

    Current user query

  • recent_turns (Array<Hash>) (defaults to: []) —

    Recent conversation turns

  • system_prompt (String, nil) (defaults to: nil) —

    Agent’s system prompt

  • read_only (Boolean) (defaults to: false) —

    When true, skips recording access on retrieved cards

Returns:

  • (Array<Hash>) —

    Messages array for LLM



101
102
103
104
105
106
107
108
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 101

def build_context(query:, recent_turns: [], system_prompt: nil, read_only: false)
  @context_builder.build(
    query: query,
    recent_turns: recent_turns,
    system_prompt: system_prompt,
    read_only: read_only,
  )
end

#compress(threshold: 1.0) ⇒ Integer

Run compression on low-exposure cards

Parameters:

  • threshold (Float) (defaults to: 1.0) —

    Exposure score threshold

Returns:

  • (Integer) —

    Number of cards compressed



132
133
134
135
136
137
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 132

def compress(threshold: 1.0)
  @compressor.compress_low_exposure(
    exposure_tracker: @exposure_tracker,
    threshold: threshold,
  )
end

#consolidate ⇒ Hash

Run consolidation (dedup, cluster updates)

Returns:

  • (Hash) —

    Summary of consolidation actions



142
143
144
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 142

def consolidate
  @consolidator.run
end

#defrag! ⇒ Hash

Run all memory defragmentation operations

Consolidates duplicates, compresses low-exposure cards, promotes frequently-accessed cards, and prunes L4 cards. Saves state after all operations complete.

Examples:

result = store.defrag!
#=> { duplicates_merged: 0, conflicts_detected: 0,
#     cards_compressed: 3, cards_promoted: 1, cards_pruned: 0 }

Returns:

  • (Hash) —

    Summary of defragmentation actions



215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 215

def defrag!
  result = {}
  start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)

  DebugLog.time("store", "defrag!") do
    # Calculate work items upfront for accurate progress tracking
    all_cards = @adapter.list_cards
    all_clusters = @adapter.list_clusters

    # Consolidate phase items
    dedup_cards = all_cards.select(&:embedding)
    conflict_types = [:constraint, :decision, :fact]
    conflict_candidates = all_cards.select { |c| conflict_types.include?(c.type) && c.embedding }

    # Compress phase items (estimate - actual may differ after consolidate)
    compress_candidates = @exposure_tracker.low_exposure_cards(threshold: 1.0)
    compress_eligible = compress_candidates.reject { |c| c.compression_level >= 4 }

    # Promote phase items
    promote_eligible = all_cards.select do |card|
      card.compression_level >= 1 &&
        card.access_count >= Configuration.instance.promotion_access_threshold
    end

    # Prune phase items
    prune_candidates = all_cards.select { |c| c.compression_level == 4 }

    # Build phase breakdown for SDK user
    phases = [
      { name: "consolidate_dedup", items: dedup_cards.size },
      { name: "consolidate_conflicts", items: conflict_candidates.size },
      { name: "consolidate_clusters", items: all_clusters.size },
      { name: "compress", items: compress_eligible.size },
      { name: "promote", items: promote_eligible.size },
      { name: "prune", items: prune_candidates.size },
    ]
    total_items = phases.sum { |p| p[:items] }

    # Emit start event with full breakdown
    EventStream.emit(
      type: "memory_defrag_start",
      total_items: total_items,
      total_cards: all_cards.size,
      total_clusters: all_clusters.size,
      phases: phases,
    )

    # Track cumulative progress across phases
    items_completed = 0

    # Phase 1: Consolidate
    consolidation = DebugLog.time("store", "consolidate") do
      @consolidator.run_with_progress(items_completed, total_items)
    end
    result[:duplicates_merged] = consolidation[:duplicates_merged]
    result[:conflicts_detected] = consolidation[:conflicts_detected]
    result[:clusters_updated] = consolidation[:clusters_updated]
    items_completed += dedup_cards.size + conflict_candidates.size + all_clusters.size

    # Phase 2: Compress
    result[:cards_compressed] = DebugLog.time("store", "compress") do
      compress_with_progress(threshold: 1.0, base_completed: items_completed, total_items: total_items)
    end
    items_completed += compress_eligible.size

    # Phase 3: Promote
    result[:cards_promoted] = DebugLog.time("store", "promote") do
      promote_with_progress(
        access_threshold: Configuration.instance.promotion_access_threshold,
        base_completed: items_completed,
        total_items: total_items,
      )
    end
    items_completed += promote_eligible.size

    # Phase 4: Prune
    result[:cards_pruned] = DebugLog.time("store", "prune") do
      prune_with_progress(base_completed: items_completed, total_items: total_items)
    end

    # Phase 5: Save
    save
  end

  # Emit completion event with summary
  elapsed_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round
  EventStream.emit(
    type: "memory_defrag_complete",
    elapsed_ms: elapsed_ms,
    duplicates_merged: result[:duplicates_merged],
    conflicts_detected: result[:conflicts_detected],
    clusters_updated: result[:clusters_updated],
    cards_compressed: result[:cards_compressed],
    cards_promoted: result[:cards_promoted],
    cards_pruned: result[:cards_pruned],
  )

  result
end

#ingest_turn(text:, turn_id:) ⇒ Array<Card>

Ingest a conversation turn into memory

Parameters:

  • text (String) —

    Full turn text

  • turn_id (String) —

    Unique turn identifier

Returns:

  • (Array<Card>) —

    Created memory cards



115
116
117
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 115

def ingest_turn(text:, turn_id:)
  @ingestion_pipeline.ingest(turn_text: text, turn_id: turn_id)
end

#load ⇒ void

This method returns an undefined value.

Load memory state from durable storage



83
84
85
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 83

def load
  DebugLog.time("store", "adapter.load") { @adapter.load }
end

#promote(access_threshold: 5) ⇒ Integer

Promote compressed cards that have been accessed frequently

Cards at L1+ with access_count above the threshold get promoted one level toward L0, rebuilding richer text from graph neighbors.

Examples:

promoted = store.promote(access_threshold: 5)

Parameters:

  • access_threshold (Integer) (defaults to: 5) —

    Minimum access count for promotion

Returns:

  • (Integer) —

    Number of cards promoted



156
157
158
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 156

def promote(access_threshold: 5)
  @compressor.promote_eligible(access_threshold: access_threshold)
end

#prune(threshold: 0.3, weights: {}) ⇒ Integer

Prune L4 cards with low retention priority

Retention priority: R = w1*importance + w2*exposure_score + w3*reuse_potential - w4*size Cards at compression level 4 with R below the threshold are deleted, along with their edges and cluster references.

Examples:

Prune with default settings

pruned = store.prune

Prune with custom threshold

pruned = store.prune(threshold: 0.5)

Parameters:

  • threshold (Float) (defaults to: 0.3) —

    Minimum retention priority to keep (default: 0.3)

  • weights (Hash) (defaults to: {}) —

    Weight parameters for retention formula

Options Hash (weights:):

  • :importance (Float) —

    Weight for card importance (default: 0.4)

  • :exposure (Float) —

    Weight for exposure score (default: 0.3)

  • :reuse (Float) —

    Weight for reuse potential (default: 0.2)

  • :size (Float) —

    Weight for size penalty (default: 0.1)

Returns:

  • (Integer) —

    Number of cards pruned



179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 179

def prune(threshold: 0.3, weights: {})
  w = {
    importance: 0.4,
    exposure: 0.3,
    reuse: 0.2,
    size: 0.1,
  }.merge(weights)

  l4_cards = @adapter.list_cards.select { |c| c.compression_level == 4 }
  return 0 if l4_cards.empty?

  pruned = 0

  l4_cards.each do |card|
    priority = retention_priority(card, w)
    next if priority >= threshold

    delete_card_with_cleanup(card.id)
    pruned += 1
  end

  pruned
end

#save ⇒ void

This method returns an undefined value.

Save memory state to durable storage



90
91
92
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 90

def save
  DebugLog.time("store", "adapter.save") { @adapter.save }
end

#search(query, top_k: 15) ⇒ Array<Card>

Search memory for relevant cards

Parameters:

  • query (String) —

    Search query

  • top_k (Integer) (defaults to: 15) —

    Number of results

Returns:

  • (Array<Card>) —

    Matching cards



124
125
126
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 124

def search(query, top_k: 15)
  @retriever.search(query, top_k: top_k)
end

#stats ⇒ Hash

Get memory statistics

Returns:

  • (Hash) —

    Memory stats



318
319
320
321
322
323
324
325
326
327
328
 
# File 'lib/swarm_sdk/v3/memory/store.rb', line 318

def stats
  cards = @adapter.list_cards
  clusters = @adapter.list_clusters

  {
    total_cards: cards.size,
    total_clusters: clusters.size,
    compression_levels: cards.group_by(&:compression_level).transform_values(&:size),
    card_types: cards.group_by(&:type).transform_values(&:size),
  }
end