Class: Ask::RAG::TextSplitter::Base

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/rag/text_splitter/base.rb

Direct Known Subclasses

Markdown, RecursiveCharacter

Instance Method Summary collapse

Constructor Details

#initialize(chunk_size: 1000, chunk_overlap: 200, length_function: :length.to_proc) ⇒ Base

Returns a new instance of Base.

Parameters:

  • chunk_size (Integer) (defaults to: 1000) —

    maximum size of each chunk

  • chunk_overlap (Integer) (defaults to: 200) —

    overlap between consecutive chunks

  • length_function (#call) (defaults to: :length.to_proc) —

    function that measures text length (default: :length)



10
11
12
13
14
 
# File 'lib/ask/rag/text_splitter/base.rb', line 10

def initialize(chunk_size: 1000, chunk_overlap: 200, length_function: :length.to_proc)
  @chunk_size = chunk_size
  @chunk_overlap = chunk_overlap
  @length_function = length_function
end

Instance Method Details

#split_documents(documents) ⇒ Array<Ask::Document>

Split documents into smaller documents.

Parameters:

  • documents (Array<Ask::Document>) —

    documents to split

Returns:

  • (Array<Ask::Document>) —

    split documents with metadata preserved



26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/ask/rag/text_splitter/base.rb', line 26

def split_documents(documents)
  documents.flat_map do |doc|
    chunks = split_text(doc.content)
    chunks.map.with_index do |chunk, idx|
      Ask::Document.new(
        content: chunk,
        metadata: doc.metadata.merge(chunk: idx)
      )
    end
  end
end

#split_text(text) ⇒ Array<String>

Split a single string into chunks.

Parameters:

  • text (String) —

    text to split

Returns:

  • (Array<String>) —

    text chunks

Raises:

  • (NotImplementedError) 


19
20
21
 
# File 'lib/ask/rag/text_splitter/base.rb', line 19

def split_text(text)
  raise NotImplementedError
end