Class: Rubino::Memory::Deduplicator

Inherits:
Object
  • Object
show all
Defined in:
lib/rubino/memory/deduplicator.rb

Overview

Prevents duplicate memories from being stored. Uses content similarity to detect duplicates.

Constant Summary collapse

SIMILARITY_THRESHOLD =

Similarity threshold (0.0 to 1.0) - above this is considered duplicate

0.85

Instance Method Summary collapse

Constructor Details

#initialize(store: nil) ⇒ Deduplicator

Returns a new instance of Deduplicator.



11
12
13
 
# File 'lib/rubino/memory/deduplicator.rb', line 11

def initialize(store: nil)
  @store = store || Store.new
end

Instance Method Details

#deduplicate_all! ⇒ Object

Removes duplicate memories, keeping the highest confidence version



22
23
24
25
26
27
28
 
# File 'lib/rubino/memory/deduplicator.rb', line 22

def deduplicate_all!
  removed = 0
  Store::VALID_KINDS.each do |kind|
    removed += deduplicate_kind(kind)
  end
  removed
end

#duplicate?(kind:, content:) ⇒ Boolean

Returns true if a similar memory already exists

Returns:

  • (Boolean) 


16
17
18
19
 
# File 'lib/rubino/memory/deduplicator.rb', line 16

def duplicate?(kind:, content:)
  existing = @store.by_kind(kind, limit: 100)
  existing.any? { |m| similar?(m[:content], content) }
end