Class: Html2rss::RequestService::BotasaurusContract

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/request_service/botasaurus_contract.rb

Overview

Maps html2rss request/response handling to botasaurus-scrape-api OpenAPI 2.0 ScrapeRequest / ScrapeSuccess / ScrapeError (sibling openapi.yaml).

Defined Under Namespace

Classes: Error, Success

Constant Summary collapse

EXECUTION_MODES =

Closed set from OpenAPI ExecutionMode.

%w[auto request browser].freeze
%w[auto get google_get google_get_bypass organic_get].freeze
ERROR_CATEGORIES =

Closed set from OpenAPI ErrorCategory.

%w[timeout challenge_block navigation_error metadata_error validation].freeze
REQUEST_OPTION_KEYS =

ScrapeRequest properties except url (html2rss supplies the target URL).

%i[
  execution_mode navigation_mode max_retries wait_for_selector wait_timeout_seconds
  scroll block_images block_images_and_css block_trackers
  wait_for_complete_page_load user_agent headers cookies window_size lang headless proxy
].freeze
WINDOW_SIZE_PROPERTIES =

OpenAPI WindowSize required keys (positive integers).

%i[width height].freeze
SCRAPE_TIMEOUT_SECONDS =

Faraday POST /scrape cap mirroring botasaurus-scrape-api total scrape wall (SCRAPE_TIMEOUT_SECONDS).

Integer(ENV.fetch('BOTASAURUS_SCRAPE_TIMEOUT_SECONDS', 45))
SCRAPE_WORK_TIMEOUT_SECONDS =

Post-boot navigate/wait budget mirrored from scrape-api (SCRAPE_WORK_TIMEOUT_SECONDS).

Integer(ENV.fetch('BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS', 30))
TRANSPORT_BUFFER_SECONDS =

Seconds added to the scrape total for client transport overhead.

2
MIN_WAIT_TIMEOUT_SECONDS =

Published wait clamp floor ([1, SCRAPE_WORK_TIMEOUT_SECONDS] in OpenAPI description).

1
MAX_WAIT_TIMEOUT_SECONDS =

Published wait clamp ceiling; YAML values above this are rejected.

SCRAPE_WORK_TIMEOUT_SECONDS
DEFAULT_WAIT_TIMEOUT_SECONDS =

OpenAPI wait_timeout_seconds default (omitted from the client payload).

[15, SCRAPE_WORK_TIMEOUT_SECONDS].min
MAX_RETRIES =

OpenAPI max_retries maximum (default 2 is applied upstream when omitted).

3
DIAGNOSTICS_KEYS =

Allowlisted ScrapeDiagnostics keys nested under diagnostics.

%w[request_id attempts strategy_used render_ms execution_tier challenge].freeze
CHALLENGE_KEYS =

Allowlisted ChallengeSignal keys nested under diagnostics.challenge.

%w[blocked detected marker].freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(url:, headers: {}, options: {}) ⇒ BotasaurusContract

Returns a new instance of BotasaurusContract.

Parameters:

  • url (Html2rss::Url) —

    canonical URL to scrape

  • headers (Hash) (defaults to: {}) —

    request headers from context

  • options (Hash) (defaults to: {}) —

    validated request.botasaurus options

Options Hash (options:):

  • :execution_mode (String)
  • :navigation_mode (String)
  • :max_retries (Integer)
  • :wait_for_selector (String)
  • :wait_timeout_seconds (Integer)
  • :scroll (Boolean)
  • :block_images (Boolean)
  • :block_images_and_css (Boolean)
  • :block_trackers (Boolean)
  • :wait_for_complete_page_load (Boolean)
  • :headless (Boolean)
  • :proxy (String)
  • :user_agent (String)
  • :window_size (Hash)
  • :lang (String)
  • :cookies (Hash)
  • :headers (Hash) 


267
268
269
270
271
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 267

def initialize(url:, headers: {}, options: {})
  @url = url
  @headers = headers
  @options = options
end

Class Method Details

.diagnostics_from(payload) ⇒ Hash{String => Object}

Returns allowlisted diagnostics (frozen).

Parameters:

  • payload (Hash{String => Object}) —

    scrape envelope

Returns:

  • (Hash{String => Object}) —

    allowlisted diagnostics (frozen)

Raises:



227
228
229
230
231
232
233
234
235
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 227

def self.diagnostics_from(payload)
  raw = payload['diagnostics']
  raise BotasaurusServiceError, 'Botasaurus scrape envelope requires diagnostics' unless raw.is_a?(Hash)
  raise BotasaurusServiceError, 'Botasaurus diagnostics require request_id' if raw['request_id'].to_s.empty?

  sliced = raw.slice(*DIAGNOSTICS_KEYS)
  sliced['challenge'] = compact_challenge(sliced['challenge'])
  sliced.compact.freeze
end

Instance Method Details

#parse_response(transport_response) ⇒ Success, Error

Parameters:

  • transport_response (Faraday::Response) —

    upstream HTTP response

Returns:

Raises:



285
286
287
288
289
290
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 285

def parse_response(transport_response)
  payload = json_object(transport_response.body.to_s)
  return Success.new(payload:) if transport_response.status == 200

  Error.new(payload:, transport_status: transport_response.status)
end

#request_payload ⇒ Hash

Returns payload for POST /scrape (explicit options plus wait cap).

Returns:

  • (Hash) —

    payload for POST /scrape (explicit options plus wait cap)



274
275
276
277
278
279
280
 
# File 'lib/html2rss/request_service/botasaurus_contract.rb', line 274

def request_payload
  payload = { url: url.to_s }.merge(filtered_options)
  forwarded_headers = merged_headers
  payload[:headers] = forwarded_headers if forwarded_headers&.any?
  cap_wait_timeout!(payload)
  payload
end