Class: Html2rss::RequestService::BotasaurusContract

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/request_service/botasaurus_contract.rb

Overview

Maps html2rss request/response handling to the botasaurus-scrape-api contract.

Defined Under Namespace

Classes: ParsedResponse

Constant Summary collapse

DEFAULT_OPTIONS =

Default Botasaurus scrape options when no explicit config is provided.

{
  execution_mode: 'auto',
  navigation_mode: 'auto',
  max_retries: 1,
  headless: false
}.freeze
OPTION_KEYS =

Allowlisted request.botasaurus keys forwarded to upstream.

%i[
  execution_mode
  navigation_mode
  max_retries
  wait_for_selector
  wait_timeout_seconds
  scroll
  scroll_to_bottom
  block_images
  block_images_and_css
  block_trackers
  wait_for_complete_page_load
  headless
  proxy
  user_agent
  window_size
  lang
  headers
  cookies
].freeze
META_KEYS =

Allowlisted upstream response keys exposed as Response#transport_meta.

%w[
  request_id strategy_used render_ms challenge_detected blocked_detected attempts error_category
  execution_tier detected_challenge
].freeze
TIGHT_BUDGET_SECONDS =

Remaining seconds at or below which Botasaurus retries are disabled.

12
BUDGET_WAIT_RESERVE_SECONDS =

Seconds reserved from remaining budget before setting wait_timeout_seconds.

2

Instance Method Summary collapse

Constructor Details

#initialize(url:, headers: {}, options: {}, remaining_timeout_seconds: nil) ⇒ BotasaurusContract

Returns a new instance of BotasaurusContract.

Parameters:

  • url (Html2rss::Url) —

    canonical URL to scrape

  • headers (Hash) (defaults to: {}) —

    request headers from context

  • options (Hash) (defaults to: {}) —

    validated request.botasaurus options

  • remaining_timeout_seconds (Numeric, nil) (defaults to: nil) —

    shared request budget remainder for clamps

Options Hash (options:):

  • :execution_mode (String)
  • :navigation_mode (String)
  • :max_retries (Integer)
  • :wait_for_selector (String)
  • :wait_timeout_seconds (Integer)
  • :block_images (Boolean)
  • :block_images_and_css (Boolean)
  • :block_trackers (Boolean)
  • :wait_for_complete_page_load (Boolean)
  • :headless (Boolean)
  • :proxy (String)
  • :user_agent (String)
  • :window_size (Array<Integer>)
  • :lang (String)
  • :cookies (Hash)
  • :headers (Hash) 


208
209
210
211
212
213
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 208

def initialize(url:, headers: {}, options: {}, remaining_timeout_seconds: nil)
  @url = url
  @headers = headers
  @options = options
  @remaining_timeout_seconds = remaining_timeout_seconds
end

Instance Method Details

#parse_response(transport_response) ⇒ ParsedResponse

Parameters:

  • transport_response (Faraday::Response) —

    upstream HTTP response

Returns:

Raises:



226
227
228
229
230
231
232
233
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 226

def parse_response(transport_response)
  payload = JSON.parse(transport_response.body.to_s)
  raise BotasaurusServiceError, 'Botasaurus response must be a JSON object' unless payload.is_a?(Hash)

  ParsedResponse.new(payload:, transport_status: transport_response.status)
rescue JSON::ParserError => error
  raise BotasaurusServiceError, "Botasaurus response JSON parse failed: #{error.message}"
end

#request_payload ⇒ Hash

Returns payload for POST /scrape (budget-clamped when remaining is known).

Returns:

  • (Hash) —

    payload for POST /scrape (budget-clamped when remaining is known)



216
217
218
219
220
221
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 216

def request_payload
  payload = DEFAULT_OPTIONS.merge(filtered_options)
  forwarded_headers = merged_headers
  payload[:headers] = forwarded_headers if forwarded_headers&.any?
  payload.merge(url: url.to_s).then { clamp_for_budget(_1) }
end