Class: Relaton::Ecma::PageFetcher

Inherits:
Object
  • Object
show all
Defined in:
lib/relaton/ecma/page_fetcher.rb

Instance Method Summary collapse

Constructor Details

#initialize ⇒ PageFetcher

Returns a new instance of PageFetcher.



4
5
6
7
 
# File 'lib/relaton/ecma/page_fetcher.rb', line 4

def initialize
  @agent = Mechanize.new
  @agent.user_agent_alias = Mechanize::AGENT_ALIASES.keys[rand(21)]
end

Instance Method Details

#get(url) ⇒ Mechanize::Page

Get page with retries

Parameters:

  • url (String) —

    url to fetch

Returns:

  • (Mechanize::Page) —

    document



16
17
18
19
20
21
22
23
24
 
# File 'lib/relaton/ecma/page_fetcher.rb', line 16

def get(url)
  3.times do |n|
    sleep n
    doc = @agent.get url
    return doc
  rescue StandardError => e
    Util.error e.message
  end
end