Class: Pikuri::VectorDb::Backend::InMemory

Inherits:
Object
  • Object
show all
Defined in:
lib/pikuri/vector_db/backend/in_memory.rb

Overview

Pure-Ruby vector store — the educational default backend, the "small enough to audit" first stop before users promote to Qdrant / Chroma for persistence. Holds a Hash from chunk id to [Chunk, vector]; #query scores cosine similarity against every stored vector, O(n) per query — fine for thousands of chunks (a notes folder), slow for millions.

Deliberately no persistence (RAM-only; reloads from sources every boot, which is why it's the teaching shape — the boot code path is the one a newcomer inspects to learn what "indexing" means) and no approximate-search index (exhaustive scan; HNSW/IVF adds complexity that teaches nothing once the cosine math is clear).

Thread-safe (the one backend that locks)

Every method runs under a reentrant Monitor — real concurrency once auto-watch is wired (the main thread queries while the Watcher thread replaces/deletes). The lock's load-bearing job is #replace_source: it holds the monitor across the delete-then-upsert so a concurrent #query never sees the zero-chunk gap. Monitor not Mutex because #replace_source re-enters via #delete_by_source + #upsert. Chroma / Qdrant need no client-side lock — the server serializes.

Cosine (not dot product) so the backend works whether or not the embedder pre-normalizes; the readable two-pass #cosine is intentional over a single-loop micro-opt — this is the file the newcomer reads.

Instance Method Summary collapse

Constructor Details

#initialize ⇒ InMemory 



35
36
37
38
39
40
41
42
43
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 35

def initialize
  # id (String) → [Chunk, vector (Array<Float>)]
  @entries = {}
  # Vector dim; +nil+ until the first +#upsert+ locks it, then enforced
  # on every +#upsert+/+#query+ — see the Backend "Vector-dim contract".
  @dim = nil
  # Reentrant — see the class header's "Thread-safe" section.
  @lock = Monitor.new
end

Instance Method Details

#count ⇒ Integer

Returns current chunk count.

Returns:

  • (Integer) —

    current chunk count.



117
118
119
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 117

def count
  @lock.synchronize { @entries.size }
end

#delete_all ⇒ void

This method returns an undefined value.

Drop every stored chunk. Used by the v1 nuke-and-reload reindex flow; the embedder dim lock is also released so a reindex with a different embedder model starts clean.



108
109
110
111
112
113
114
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 108

def delete_all
  @lock.synchronize do
    @entries.clear
    @dim = nil
  end
  nil
end

#delete_by_source(source) ⇒ void

This method returns an undefined value.

Remove every chunk whose source matches. The scoped counterpart to #delete_all — drops one document's chunks without touching the rest. No-op (and no error) when the source isn't present. The dim lock is left intact: unlike #delete_all, a per-source delete doesn't imply an embedder change.

Parameters:

  • source (String) —

    the Chunk#source to purge, e.g. "notes/cooking.md".



131
132
133
134
135
136
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 131

def delete_by_source(source)
  @lock.synchronize do
    @entries.reject! { |_id, (chunk, _vector)| chunk.source == source }
  end
  nil
end

#query(vector:, top_k:) ⇒ Array<Backend::Result>

Cosine-similarity nearest neighbor search. Returns the top-k Results in descending score order; empty array when the store has no entries.

Parameters:

  • vector (Array<Float>) —

    query vector; must match the stored vector dim.

  • top_k (Integer) —

    number of results to return; must be positive.

Returns:

Raises:

  • (ArgumentError) —

    on top_k <= 0 or query-vector dim mismatch.



86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 86

def query(vector:, top_k:)
  raise ArgumentError, "top_k must be positive (got #{top_k})" if top_k <= 0

  @lock.synchronize do
    return [] if @entries.empty?

    if vector.size != @dim
      raise ArgumentError, "query vector dim #{vector.size}, stored dim #{@dim}"
    end

    scored = @entries.values.map do |chunk, stored|
      Result.new(chunk: chunk, score: cosine(vector, stored))
    end
    scored.sort_by { |r| -r.score }.first(top_k)
  end
end

#replace_source(source:, chunks:, vectors:) ⇒ void

This method returns an undefined value.

Atomically replace all chunks for one source (delete-then-upsert under a single monitor hold — the point: a concurrent #query sees the old or new chunks, never the empty gap). The incremental-reindex unit (see Indexer#reindex_file!).

Parameters:

  • source (String) —

    the Chunk#source being replaced.

  • chunks (Array<Chunk>) —

    the new chunk set (each chunk.source should equal source).

  • vectors (Array<Array<Float>>) —

    parallel to chunks.

Raises:

  • (ArgumentError) —

    on empty input, length or vector-dim mismatch.



149
150
151
152
153
154
155
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 149

def replace_source(source:, chunks:, vectors:)
  @lock.synchronize do
    delete_by_source(source)
    upsert(chunks: chunks, vectors: vectors)
  end
  nil
end

#source_indexed?(source) ⇒ Boolean

Is source in the corpus? The scoped membership test behind Tools::Read's gate — a short-circuiting scan rather than building the whole #sources_with_hashes map just to read one key. See the Backend protocol yardoc.

Parameters:

Returns:

  • (Boolean) —

    true if at least one chunk has this source.



181
182
183
184
185
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 181

def source_indexed?(source)
  @lock.synchronize do
    @entries.each_value.any? { |chunk, _vector| chunk.source == source }
  end
end

#sources_with_hashes ⇒ Hash{String => String, nil}

The boot-sweep reference: each indexed source → the content hash on its chunks, which Indexer#reconcile_plan diffs against disk. A chunk with no hash metadata maps to nil, which the diff treats as "changed" and reindexes — self-healing.

Returns:

  • (Hash{String => String, nil}) —

    source → content hash. Empty when nothing is indexed (the InMemory case at every boot).



164
165
166
167
168
169
170
171
172
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 164

def sources_with_hashes
  @lock.synchronize do
    result = {}
    @entries.each_value do |chunk, _vector|
      result[chunk.source] ||= chunk.metadata[:hash]
    end
    result
  end
end

#upsert(chunks:, vectors:) ⇒ void

This method returns an undefined value.

Insert-or-replace by chunk.id. Parallel arrays of equal length; raises on empty input or length mismatch. Vector dimension is locked at first upsert; raises on any subsequent vector of a different dim.

Parameters:

  • chunks (Array<Chunk>)
  • vectors (Array<Array<Float>>)

Raises:

  • (ArgumentError) —

    on empty input, length mismatch, or vector-dim mismatch.



55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/pikuri/vector_db/backend/in_memory.rb', line 55

def upsert(chunks:, vectors:)
  raise ArgumentError, 'upsert called with empty chunks/vectors' if chunks.empty?
  if chunks.size != vectors.size
    raise ArgumentError, "size mismatch: #{chunks.size} chunks vs #{vectors.size} vectors"
  end

  @lock.synchronize do
    expected = @dim || vectors.first.size
    vectors.each_with_index do |v, i|
      next if v.size == expected

      raise ArgumentError, "vector #{i} has dim #{v.size}, expected #{expected}"
    end
    @dim ||= expected

    chunks.zip(vectors).each { |chunk, vector| @entries[chunk.id] = [chunk, vector] }
  end
  nil
end