Class: Html2rss::RequestService::BotasaurusContract

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/request_service/botasaurus_contract.rb

Overview

Maps html2rss request/response handling to the botasaurus-scrape-api contract.

Defined Under Namespace

Classes: ParsedResponse

Constant Summary collapse

DEFAULT_OPTIONS =

Default Botasaurus scrape options when no explicit config is provided.

{
  navigation_mode: 'auto',
  max_retries: 1,
  headless: false
}.freeze
OPTION_KEYS =

Allowlisted request.botasaurus keys forwarded to upstream.

%i[
  navigation_mode
  max_retries
  wait_for_selector
  wait_timeout_seconds
  block_images
  block_images_and_css
  wait_for_complete_page_load
  headless
  proxy
  user_agent
  window_size
  lang
].freeze
META_KEYS =

Allowlisted upstream response keys exposed as Response#transport_meta.

%w[
  request_id strategy_used render_ms challenge_detected blocked_detected attempts error_category
].freeze
TIGHT_BUDGET_SECONDS =

Remaining seconds at or below which Botasaurus retries are disabled.

12
BUDGET_WAIT_RESERVE_SECONDS =

Seconds reserved from remaining budget before setting wait_timeout_seconds.

2

Instance Method Summary collapse

Constructor Details

#initialize(url:, options: {}, remaining_timeout_seconds: nil) ⇒ BotasaurusContract

Returns a new instance of BotasaurusContract.

Parameters:

  • url (Html2rss::Url) —

    canonical URL to scrape

  • options (Hash) (defaults to: {}) —

    validated request.botasaurus options

  • remaining_timeout_seconds (Numeric, nil) (defaults to: nil) —

    shared request budget remainder for clamps

Options Hash (options:):

  • :navigation_mode (String)
  • :max_retries (Integer)
  • :wait_for_selector (String)
  • :wait_timeout_seconds (Integer)
  • :block_images (Boolean)
  • :block_images_and_css (Boolean)
  • :wait_for_complete_page_load (Boolean)
  • :headless (Boolean)
  • :proxy (String)
  • :user_agent (String)
  • :window_size (Array<Integer>)
  • :lang (String) 


139
140
141
142
143
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 139

def initialize(url:, options: {}, remaining_timeout_seconds: nil)
  @url = url
  @options = options
  @remaining_timeout_seconds = remaining_timeout_seconds
end

Instance Method Details

#parse_response(transport_response) ⇒ ParsedResponse

Parameters:

  • transport_response (Faraday::Response) —

    upstream HTTP response

Returns:

Raises:



153
154
155
156
157
158
159
160
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 153

def parse_response(transport_response)
  payload = JSON.parse(transport_response.body.to_s)
  raise BotasaurusConnectionFailed, 'Botasaurus response must be a JSON object' unless payload.is_a?(Hash)

  ParsedResponse.new(payload:, transport_status: transport_response.status)
rescue JSON::ParserError => error
  raise BotasaurusConnectionFailed, "Botasaurus response JSON parse failed: #{error.message}"
end

#request_payload ⇒ Hash

Returns payload for POST /scrape (budget-clamped when remaining is known).

Returns:

  • (Hash) —

    payload for POST /scrape (budget-clamped when remaining is known)



146
147
148
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 146

def request_payload
  DEFAULT_OPTIONS.merge(filtered_options).merge(url: url.to_s).then { clamp_for_budget(_1) }
end