Class: Rubino::Memory::Deduplicator

Inherits:
Object
  • Object
show all
Defined in:
lib/rubino/memory/deduplicator.rb

Overview

Prevents duplicate memories from being stored. Uses content similarity to detect duplicates.

Scope is read-then-write WITHIN one extraction, not a write-time uniqueness constraint (#49): #duplicate? reads existing rows and Store#create inserts without a unique index, so two concurrent rubino instances that extract the SAME fact in the same instant can each pass the check and write one row —two identical rows, no data loss. This matches the field: mem0 likewise dedups per-extraction (exact/MD5 + similarity) with no cross-writer locking (mem0ai/mem0#4896). #deduplicate_all! can collapse any such pair on demand. The same-instant cross-instance collision is a rare, benign edge — by design, not a bug to gate every write on a lock.

Constant Summary collapse

SIMILARITY_THRESHOLD =

Similarity threshold (0.0 to 1.0) - above this is considered duplicate

0.85

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(store: nil) ⇒ Deduplicator

Returns a new instance of Deduplicator.



33
34
35
 
# File 'lib/rubino/memory/deduplicator.rb', line 33

def initialize(store: nil)
  @store = store || Store.new
end

Class Method Details

.normalize_verbatim(text) ⇒ Object

Normalize a fact for an EXACT-verbatim compare: collapse runs of whitespace to one space, strip the ends, and case-fold (#Y4). Two facts with the same normalized form are byte-equal-enough to be one fact, so a second save is a no-op. This is distinct from the 0.85 Jaccard near-dup (which a word-reordering rephrase can satisfy but #93-F4 showed misses the trivial “saved twice” repeat after the live set churns) and from the cross-instance semantic merge (#49). The single source of truth for what “the same fact” means at the write seam, shared by every backend.



29
30
31
 
# File 'lib/rubino/memory/deduplicator.rb', line 29

def self.normalize_verbatim(text)
  text.to_s.gsub(/\s+/, " ").strip.downcase
end

Instance Method Details

#deduplicate_all! ⇒ Object

Removes duplicate memories, keeping the highest confidence version



44
45
46
47
48
49
50
 
# File 'lib/rubino/memory/deduplicator.rb', line 44

def deduplicate_all!
  removed = 0
  Store::VALID_KINDS.each do |kind|
    removed += deduplicate_kind(kind)
  end
  removed
end

#duplicate?(kind:, content:) ⇒ Boolean

Returns true if a similar memory already exists

Returns:

  • (Boolean) 


38
39
40
41
 
# File 'lib/rubino/memory/deduplicator.rb', line 38

def duplicate?(kind:, content:)
  existing = @store.by_kind(kind, limit: 100)
  existing.any? { |m| similar?(m[:content], content) }
end