Class: Pikuri::VectorDb::Chunker::FixedWindow

Inherits:
Object
  • Object
show all
Defined in:
lib/pikuri/vector_db/chunker/fixed_window.rb

Overview

Sliding-window chunker: splits text on whitespace into words, then walks forward emitting ~+size+-token chunks, each after the first re-including overlap tokens of the previous chunk's tail so an answer straddling a boundary stays intact in at least one chunk (standard RAG practice).

Whitespace-split gives word boundaries for Western-European languages — good enough for any text FileType.read_as_text flattens to prose; CJK (no whitespace) degrades to one-huge-unit chunks, a documented limitation.

Per-chunk tokenizer cost: the greedy "add a word, re-count" algorithm calls tokenizer.count O(n_words × chunks) times — negligible for Tokenizer::CharHeuristic, but one HTTP round-trip each for Tokenizer::LlamaServer, so indexing a large corpus takes minutes. A one-time boot/reindex cost, accepted for v1.

Forward-progress guard: the constructor rejects overlap >= size, and the inner loop always advances at least one word — termination holds even if a pathological tokenizer reports misleading counts.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(size:, overlap: 0, tokenizer: Tokenizer::CharHeuristic.new) ⇒ FixedWindow

Parameters:

  • size (Integer) —

    target token count per chunk. Must be positive. Common values: 256, 512, 1024 — pick to match the embedder's context (e.g. 512 for bge-small-en-v1.5).

  • overlap (Integer) (defaults to: 0) —

    tokens of overlap between adjacent chunks. Must be >= 0 and strictly less than size. Common values: ~10% of size.

  • tokenizer (#count) (defaults to: Tokenizer::CharHeuristic.new) —

    a Tokenizer; anything responding to count(text) -> Integer. Defaults to Tokenizer::CharHeuristic (zero-dep, ~4-chars-per-token approximation).

Raises:

  • (ArgumentError) —

    on invalid size or overlap.



46
47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/pikuri/vector_db/chunker/fixed_window.rb', line 46

def initialize(size:, overlap: 0, tokenizer: Tokenizer::CharHeuristic.new)
  raise ArgumentError, "size must be positive (got #{size})" if size <= 0
  raise ArgumentError, "overlap must be >= 0 (got #{overlap})" if overlap.negative?
  if overlap >= size
    raise ArgumentError,
          "overlap (#{overlap}) must be strictly less than size (#{size}) " \
          "— the sliding window would not advance"
  end

  @size = size
  @overlap = overlap
  @tokenizer = tokenizer
end

Instance Attribute Details

#overlap ⇒ Integer (readonly)

Returns tokens of overlap between adjacent chunks.

Returns:

  • (Integer) —

    tokens of overlap between adjacent chunks.



31
32
33
 
# File 'lib/pikuri/vector_db/chunker/fixed_window.rb', line 31

def overlap
  @overlap
end

#size ⇒ Integer (readonly)

Returns target token count per chunk.

Returns:

  • (Integer) —

    target token count per chunk.



27
28
29
 
# File 'lib/pikuri/vector_db/chunker/fixed_window.rb', line 27

def size
  @size
end

Instance Method Details

#chunk(text) ⇒ Array<String>

Chunk text into approximately +size+-token windows with +overlap+-token tail repeats. Empty / whitespace-only input returns [].

Parameters:

  • text (String)

Returns:

  • (Array<String>) —

    non-empty chunks, in source order. May be empty.



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
 
# File 'lib/pikuri/vector_db/chunker/fixed_window.rb', line 67

def chunk(text)
  words = text.split
  return [] if words.empty?

  chunks = []
  start = 0
  while start < words.length
    finish = find_chunk_end(words, start)
    chunks << words[start...finish].join(' ')

    break if finish >= words.length

    start = find_next_start(words, start, finish)
  end

  chunks
end