Class: ActiveRecordVector::Chunker

Inherits:
Object
  • Object
show all
Defined in:
lib/active_record_vector/chunker.rb

Overview

Document text chunking utility for RAG (Retrieval-Augmented Generation) pipelines. Splits long texts into overlapping chunks for embedding generation.

Constant Summary collapse

DEFAULT_SEPARATORS = 
["\n\n", "\n", ". ", " ", ""].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(chunk_size: 1000, chunk_overlap: 200, separators: DEFAULT_SEPARATORS) ⇒ Chunker

Returns a new instance of Chunker.



11
12
13
14
15
 
# File 'lib/active_record_vector/chunker.rb', line 11

def initialize(chunk_size: 1000, chunk_overlap: 200, separators: DEFAULT_SEPARATORS)
  @chunk_size = chunk_size
  @chunk_overlap = chunk_overlap
  @separators = separators
end

Instance Attribute Details

#chunk_overlap ⇒ Object (readonly)

Returns the value of attribute chunk_overlap.



7
8
9
 
# File 'lib/active_record_vector/chunker.rb', line 7

def chunk_overlap
  @chunk_overlap
end

#chunk_size ⇒ Object (readonly)

Returns the value of attribute chunk_size.



7
8
9
 
# File 'lib/active_record_vector/chunker.rb', line 7

def chunk_size
  @chunk_size
end

#separators ⇒ Object (readonly)

Returns the value of attribute separators.



7
8
9
 
# File 'lib/active_record_vector/chunker.rb', line 7

def separators
  @separators
end

Class Method Details

.split(text, chunk_size: 1000, chunk_overlap: 200) ⇒ Object

Convenience class method



18
19
20
 
# File 'lib/active_record_vector/chunker.rb', line 18

def self.split(text, chunk_size: 1000, chunk_overlap: 200)
  new(chunk_size: chunk_size, chunk_overlap: chunk_overlap).split(text)
end

Instance Method Details

#split(text) ⇒ Object

Split text into array of chunk strings



23
24
25
26
27
28
29
 
# File 'lib/active_record_vector/chunker.rb', line 23

def split(text)
  return [] if text.nil? || text.strip.empty?
  return [text.strip] if text.length <= @chunk_size

  splits = split_text(text, @separators)
  merge_splits(splits)
end