Class: Ask::RAG::Loader::HTML

Inherits:
Base
  • Object
show all
Defined in:
lib/ask/rag/loader/html.rb

Overview

Loads an HTML file and extracts text content using Nokogiri. Strips script, style, and nav elements by default.

Instance Method Summary collapse

Methods inherited from Base

#lazy_load

Constructor Details

#initialize(elements_to_remove: %w[script style nav footer header])) ⇒ HTML

Returns a new instance of HTML.

Parameters:

  • elements_to_remove (Array<String>) (defaults to: %w[script style nav footer header])) —

    CSS selectors to strip



10
11
12
 
# File 'lib/ask/rag/loader/html.rb', line 10

def initialize(elements_to_remove: %w[script style nav footer header])
  @elements_to_remove = elements_to_remove
end

Instance Method Details

#load(path) ⇒ Object 



14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 
# File 'lib/ask/rag/loader/html.rb', line 14

def load(path)
  content = File.read(path)
  doc = Nokogiri::HTML.parse(content)

  @elements_to_remove.each do |selector|
    doc.css(selector).each(&:remove)
  end

  text = doc.at_xpath("//body")&.text || doc.text
  text = text.gsub(/\s+/, " ").strip

  [Ask::Document.new(
    content: text,
    metadata: { source: path, format: "html" }
  )]
end