Class: RCrewAI::Knowledge::UrlSource

Inherits:
Source
  • Object
show all
Defined in:
lib/rcrewai/knowledge/sources.rb

Instance Method Summary collapse

Constructor Details

#initialize(url, fetcher: nil) ⇒ UrlSource

Returns a new instance of UrlSource.



61
62
63
64
65
 
# File 'lib/rcrewai/knowledge/sources.rb', line 61

def initialize(url, fetcher: nil)
  super()
  @url = url
  @fetcher = fetcher
end

Instance Method Details

#read ⇒ Object 



67
68
69
70
71
72
73
 
# File 'lib/rcrewai/knowledge/sources.rb', line 67

def read
  html = @fetcher ? @fetcher.call(@url) : fetch(@url)
  require 'nokogiri'
  doc = Nokogiri::HTML(html)
  doc.search('script, style').remove
  doc.text.gsub(/\s+/, ' ').strip
end