Class: Tina4::Context

Inherits:
Object
  • Object
show all
Defined in:
lib/tina4/context.rb,
lib/tina4/context/chunker.rb

Defined Under Namespace

Modules: Chunker

Constant Summary collapse

CODE_EXTS =

File classification — mirrors the Python port's repo walk.

%w[.py .php .js .mjs .ts .rb .pas .dpr .dpk .inc .dfm .fmx].freeze
DOC_EXTS = 
%w[.md .txt .rst .twig .html].freeze
CONFIG_EXTS =

deploy/CLI/env answers live in config files, not sources — chunk as code (line windows), since sentence chunking shreds YAML/Dockerfiles.

%w[.toml .yml .yaml].freeze
SPECIAL_FILES = 
%w[dockerfile makefile docker-compose.yml package.json
composer.json gemfile .env.example .env.sample].freeze
SKIP_DIRS =

Same dirs the Python port skips, plus Tina4 runtime dirs that hold no source of truth (our own index/backups, session blobs, logs).

%w[.git __pycache__ node_modules vendor dist build coverage
.idea .venv venv .pytest_cache .tina4 sessions logs].freeze
DEF_STOP =

Generic question/code vocabulary that never NAMES a symbol — kept small.

%w[the and what how does can which where when who why list all
available module class function functions method methods def
get set new return import from with that this are is was for
into use used].to_set.freeze
DEF_KW =

A chunk that DEFINES a queried symbol ("def get_token", "class Widget", "module Auth") should out-rank one that merely USES it. Matched against the folded body.

'(?:async def|def|class|module|function|fn|func|interface|trait)'

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(path = "./.tina4/context.db", fts5_check: nil) ⇒ Context

path is the on-disk index file (its parent dir is created). fts5_check overrides FTS5 detection (used by tests to exercise the graceful- degradation path); defaults to a real probe of this build.



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
 
# File 'lib/tina4/context.rb', line 64

def initialize(path = "./.tina4/context.db", fts5_check: nil)
  @path = path.to_s
  @lock = Mutex.new
  @conn = nil
  @root = nil # set by index_root; reindex_file relabels against it
  check = fts5_check || -> { self.class.fts5_available? }
  @available = !!check.call
  return unless @available

  require "sqlite3"
  parent = File.dirname(@path)
  FileUtils.mkdir_p(parent) unless parent.empty? || parent == "." || File.directory?(parent)
  # One connection guarded by a Mutex — the dev-MCP reload hook may run on a
  # Puma worker thread, so access is serialized rather than same-thread-only.
  @conn = SQLite3::Database.new(@path)
  ensure_table
end

Instance Attribute Details

#available ⇒ Object (readonly)

Returns the value of attribute available.



59
60
61
 
# File 'lib/tina4/context.rb', line 59

def available
  @available
end

#path ⇒ Object (readonly)

Returns the value of attribute path.



59
60
61
 
# File 'lib/tina4/context.rb', line 59

def path
  @path
end

#root ⇒ Object (readonly)

Returns the value of attribute root.



59
60
61
 
# File 'lib/tina4/context.rb', line 59

def root
  @root
end

Class Method Details

.chunks_for(label, text) ⇒ Object

── indexing ──────────────────────────────────────────────── [index, chunk_text] pairs — code/config files on def/class boundaries, docs as prose.



124
125
126
127
128
129
130
131
132
 
# File 'lib/tina4/context.rb', line 124

def self.chunks_for(label, text)
  ext = File.extname(label).downcase
  special = SPECIAL_FILES.include?(File.basename(label).downcase)
  if CODE_EXTS.include?(ext) || CONFIG_EXTS.include?(ext) || special
    Chunker.chunk_code(text, path: label)
  else
    Chunker.chunk_text(text)
  end
end

.clear_shared_contexts! ⇒ Object

Test/teardown seam — forget every shared Context.



400
401
402
403
 
# File 'lib/tina4/context.rb', line 400

def clear_shared_contexts!
  @shared_contexts.each_value { |c| c.close rescue nil }
  @shared_contexts = {}
end

.db_key(db = nil) ⇒ Object 



376
377
378
379
 
# File 'lib/tina4/context.rb', line 376

def db_key(db = nil)
  base = db ? db.to_s : File.join(Dir.pwd, ".tina4", "context.db")
  File.expand_path(base)
end

.default_context(root: nil, db: nil) ⇒ Object

Get (or create) the process-wide Context at db (default /.tina4/context.db). If root is given and the index is empty, builds it once. This is what code_search uses so the reload hook can keep the SAME index fresh.



385
386
387
388
389
390
 
# File 'lib/tina4/context.rb', line 385

def default_context(root: nil, db: nil)
  key = db_key(db)
  ctx = (@shared_contexts[key] ||= new(key))
  ctx.index_root(root) if !root.nil? && ctx.available && ctx.empty?
  ctx
end

.eligible?(filename) ⇒ Boolean

True if a file should be indexed (the per-file filter used by both index_root and reindex_file). Directory skipping is handled separately.

Returns:

  • (Boolean) 


167
168
169
170
171
172
173
174
 
# File 'lib/tina4/context.rb', line 167

def self.eligible?(filename)
  fn = filename.downcase
  return false if fn.end_with?(".min.js")

  ext = File.extname(fn)
  CODE_EXTS.include?(ext) || DOC_EXTS.include?(ext) ||
    CONFIG_EXTS.include?(ext) || SPECIAL_FILES.include?(fn)
end

.existing_context(db: nil) ⇒ Object

Return the already-created shared Context for db (or nil). Used by the reload hook so a file change reindexes an EXISTING index but never creates one on its own (nothing to keep fresh until code_search runs).



395
396
397
 
# File 'lib/tina4/context.rb', line 395

def existing_context(db: nil)
  @shared_contexts[db_key(db)]
end

.fts5_available? ⇒ Boolean

── FTS5 availability ─────────────────────────────────────── Whether this Ruby's sqlite3 build supports FTS5 (a real in-memory probe).

Returns:

  • (Boolean) 


84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/tina4/context.rb', line 84

def self.fts5_available?
  require "sqlite3"
  db = SQLite3::Database.new(":memory:")
  begin
    db.execute("CREATE VIRTUAL TABLE _probe USING fts5(x)")
    true
  rescue SQLite3::Exception
    false
  ensure
    db.close
  end
rescue LoadError, StandardError
  false
end

.testlike?(path) ⇒ Boolean

Returns:

  • (Boolean) 


242
243
244
245
 
# File 'lib/tina4/context.rb', line 242

def self.testlike?(path)
  s = (path || "").downcase
  ["/test", "test/", "test_", "_test.", ".test.", "/example", "example/"].any? { |p| s.include?(p) }
end

Instance Method Details

#close ⇒ Object 



315
316
317
318
319
320
321
322
 
# File 'lib/tina4/context.rb', line 315

def close
  return if @conn.nil?

  @lock.synchronize do
    @conn.close
    @conn = nil
  end
end

#count ⇒ Object

── misc ────────────────────────────────────────────────────



305
306
307
308
309
 
# File 'lib/tina4/context.rb', line 305

def count
  return 0 unless @available

  @lock.synchronize { @conn.execute("SELECT count(*) FROM chunks").first.first }
end

#empty? ⇒ Boolean

Returns:

  • (Boolean) 


311
312
313
 
# File 'lib/tina4/context.rb', line 311

def empty?
  count.zero?
end

#index_path(file, label: nil) ⇒ Object

UPSERT one file into the index: delete this path's existing chunks, re-chunk the current contents, insert. A single file save re-indexes just that file. label is the stored/citation path (defaults to file) and MUST be stable across calls for the same file so the delete targets the right rows. Returns rows inserted.



139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
 
# File 'lib/tina4/context.rb', line 139

def index_path(file, label: nil)
  return 0 unless @available

  stored = as_text(label.nil? ? file : label)
  begin
    text = File.read(file, encoding: "UTF-8", invalid: :replace, undef: :replace, replace: "")
  rescue SystemCallError
    return 0
  end
  rows = self.class.chunks_for(stored, text).map do |i, chunk|
    ["#{stored}:#{i}", stored, chunk, Chunker.fold(chunk)]
  end
  @lock.synchronize do
    @conn.execute("DELETE FROM chunks WHERE path = ?", [stored])
    unless rows.empty?
      stmt = @conn.prepare("INSERT INTO chunks(cid, path, raw, body) VALUES (?, ?, ?, ?)")
      begin
        rows.each { |r| stmt.execute(r) }
      ensure
        stmt.close
      end
    end
  end
  rows.length
end

#index_root(root) ⇒ Object

Walk root, indexing every eligible file (skips vendor/build/runtime dirs). Paths are stored RELATIVE to root for clean citations. Records root so reindex_file can relabel a changed file consistently. Returns the total number of chunks inserted.



180
181
182
183
184
185
186
187
188
189
190
191
 
# File 'lib/tina4/context.rb', line 180

def index_root(root)
  return 0 unless @available

  root = canonical_dir(root)
  @root = root
  total = 0
  walk(root) do |full|
    rel = Pathname.new(full).relative_path_from(Pathname.new(root)).to_s
    total += index_path(full, label: rel)
  end
  total
end

#reindex_file(changed_path) ⇒ Object

Re-index a single changed file into the LIVE index — the hook the dev reload trigger (POST /__dev/api/reload) calls so code_search tracks edits without a rebuild. Resolves changed_path against the indexed root, then: outside root / under a skip-or-dot dir / ineligible -> skip (-1); deleted -> drop its chunks (0); otherwise UPSERT (rows). No-op (-1) until index_root has run (nothing to keep fresh yet).



199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/tina4/context.rb', line 199

def reindex_file(changed_path)
  return -1 unless @available && @root

  p = changed_path.to_s
  # The reload trigger reports paths relative to the project root (cwd during
  # `tina4 serve`); the index root may be a subdir like src/.
  p = File.join(Dir.pwd, p) unless Pathname.new(p).absolute?
  resolved = canonical_path(p)
  begin
    rel_pn = Pathname.new(resolved).relative_path_from(Pathname.new(@root))
  rescue ArgumentError
    return -1 # different mount/drive — outside the indexed root
  end
  parts = rel_pn.each_filename.to_a
  return -1 if parts.first == ".." # outside the indexed root
  return -1 if parts.any? { |pt| SKIP_DIRS.include?(pt) }
  return -1 if parts[0...-1].any? { |pt| pt.start_with?(".") }
  return -1 unless self.class.eligible?(rel_pn.basename.to_s)

  stored = as_text(rel_pn.to_s)
  unless File.exist?(resolved) # deleted → drop its chunks
    @lock.synchronize { @conn.execute("DELETE FROM chunks WHERE path = ?", [stored]) }
    return 0
  end
  index_path(resolved, label: stored)
end

#reset ⇒ Object

Drop and recreate the index (full rebuild starting point).



112
113
114
115
116
117
118
119
 
# File 'lib/tina4/context.rb', line 112

def reset
  return unless @available

  @lock.synchronize do
    @conn.execute("DROP TABLE IF EXISTS chunks")
    ensure_table
  end
end

#search(query, k: 5) ⇒ Object

Return the top-k chunks as [{ path:, score:, snippet: }], ranked by bm25() then reordered with two stable, proven passes:

- source-over-tests: a test that merely mentions a symbol sinks below the
source that defines it (skipped when the query is about tests);
- definition-first: a chunk that DEFINES a queried symbol rises above
chunks that only use it.

Score is a higher-is-better float (sqlite's bm25 sign flipped).



263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
 
# File 'lib/tina4/context.rb', line 263

def search(query, k: 5)
  return [] unless @available

  expr = match_expr(query)
  return [] if expr.nil?

  pool_n = [k * 3, 15].max
  rows = @lock.synchronize do
    @conn.execute(
      "SELECT path, raw, body, bm25(chunks) AS s FROM chunks " \
      "WHERE chunks MATCH ? ORDER BY s LIMIT ?",
      [expr, pool_n]
    )
  end
  return [] if rows.empty?

  # candidate symbol names from the query (drop generic vocab).
  symbols = Chunker.terms(query).select { |t| t.length >= 3 && !DEF_STOP.include?(t) }.uniq
  about_tests = query.downcase.include?("test")

  ordered = rows.each_with_index.sort_by do |row, idx|
    path, _raw, body, = row
    is_test = (!about_tests && self.class.testlike?(path)) ? 1 : 0
    is_def  = defines?(body, symbols) ? 0 : 1
    [is_test, is_def, idx] # bm25 order (idx) breaks ties
  end

  ordered.first(k).map do |row, _idx|
    path, raw, _body, s = row
    { path: path, score: (-s.to_f).round(6), snippet: snippet(raw) }
  end
end