Class: Ask::RAG::TextSplitter::RecursiveCharacter

Inherits:
Base
  • Object
show all
Defined in:
lib/ask/rag/text_splitter/recursive_character.rb

Overview

Splits text recursively by descending separator length.

The default separators try paragraph breaks, then line breaks, then sentences, then words, then characters. This produces semantically coherent chunks that fit within chunk_size.

Examples:

splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(
  chunk_size: 500, chunk_overlap: 50
)
chunks = splitter.split_text(long_text)

Instance Method Summary collapse

Methods inherited from Base

#split_documents

Constructor Details

#initialize(separators: ["\n\n", "\n", ". ", " ", ""], **options) ⇒ RecursiveCharacter

Returns a new instance of RecursiveCharacter.

Parameters:

  • separators (Array<String>) (defaults to: ["\n\n", "\n", ". ", " ", ""]) —

    ordered list of separators to try

  • chunk_size (Integer) —

    max chunk size

  • chunk_overlap (Integer) —

    overlap between chunks



22
23
24
25
 
# File 'lib/ask/rag/text_splitter/recursive_character.rb', line 22

def initialize(separators: ["\n\n", "\n", ". ", " ", ""], **options)
  @separators = separators
  super(**options)
end

Instance Method Details

#split_text(text) ⇒ Object 



27
28
29
30
 
# File 'lib/ask/rag/text_splitter/recursive_character.rb', line 27

def split_text(text)
  splits = split_with_separators(text, @separators)
  merge_splits(splits)
end