Class: RCrewAI::Knowledge::Chunker

Inherits:
Object
  • Object
show all
Defined in:
lib/rcrewai/knowledge/chunker.rb

Overview

Splits text into fixed-size, overlapping character windows. Overlap keeps context from spilling across chunk boundaries during retrieval.

Instance Method Summary collapse

Constructor Details

#initialize(chunk_size: 1000, overlap: 100) ⇒ Chunker

Returns a new instance of Chunker.

Raises:

  • (ArgumentError) 


8
9
10
11
12
13
 
# File 'lib/rcrewai/knowledge/chunker.rb', line 8

def initialize(chunk_size: 1000, overlap: 100)
  raise ArgumentError, 'overlap must be smaller than chunk_size' if overlap >= chunk_size

  @chunk_size = chunk_size
  @overlap = overlap
end

Instance Method Details

#chunk(text) ⇒ Object 



15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
# File 'lib/rcrewai/knowledge/chunker.rb', line 15

def chunk(text)
  text = text.to_s
  return [] if text.empty?
  return [text] if text.length <= @chunk_size

  chunks = []
  start = 0
  step = @chunk_size - @overlap
  while start < text.length
    chunks << text[start, @chunk_size]
    start += step
  end
  chunks
end