Module: Html2rss::RequestService::BlockedSurface

Defined in:
lib/html2rss/request_service/blocked_surface.rb

Overview

Shared anti-bot/interstitial signatures used by request and auto-source flows.

This module centralizes signature matching so request-time guards and auto-source surface classification stay consistent.

Constant Summary collapse

INTERSTITIAL_SIGNATURES =

Known interstitial fingerprints used to detect blocked or anti-bot surfaces.

[
  {
    key: :cloudflare_interstitial,
    min_matches: 2,
    patterns: [
      %r{<title>\s*just a moment\.\.\.\s*</title>}i,
      /checking your browser before accessing/i,
      /please (?:enable|turn on) javascript and cookies/i,
      %r{cdn-cgi/challenge-platform}i,
      /cloudflare ray id/i
    ],
    message: 'Blocked surface detected: Cloudflare anti-bot interstitial page. ' \
             'Target a direct listing URL, configure BOTASAURUS_SCRAPER_URL, ' \
             'or run from an environment that can complete anti-bot checks.'
  },
  {
    key: :datadome_interstitial,
    min_matches: 2,
    patterns: [
      /captcha-delivery\.com/,
      /DataDome/i,
      /interstitial/i
    ],
    message: 'Blocked surface detected: DataDome anti-bot challenge page. ' \
             'Target a direct listing URL, or scrape via a session with resolved DataDome cookies.'
  },
  {
    key: :vercel_security_checkpoint,
    min_matches: 1,
    patterns: [
      /Vercel Security Checkpoint/i,
      %r{vercel\.com/security}i,
      /checking the security/i
    ],
    message: 'Blocked surface detected: Vercel Security Checkpoint. ' \
             'This site is a JS-rendered SPA behind Vercel edge protection. ' \
             'Configure BOTASAURUS_SCRAPER_URL or target a direct listing URL.'
  }
].freeze

Class Method Summary collapse

Class Method Details

.interstitial?(body) ⇒ Boolean

Returns true when body matches a known interstitial signature.

Parameters:

  • body (String, nil) —

    response body candidate

Returns:

  • (Boolean) —

    true when body matches a known interstitial signature



65
66
67
 
# File 'lib/html2rss/request_service/blocked_surface.rb', line 65

def self.interstitial?(body)
  !interstitial_signature_for(body).nil?
end

.interstitial_signature_for(body) ⇒ Hash?

Returns the first matching interstitial signature for the provided body.

Parameters:

  • body (String, nil) —

    response body candidate

Returns:

  • (Hash, nil) —

    signature hash when matched, otherwise nil



57
58
59
60
 
# File 'lib/html2rss/request_service/blocked_surface.rb', line 57

def self.interstitial_signature_for(body)
  normalized_body = normalize_body(body)
  INTERSTITIAL_SIGNATURES.find { |signature| interstitial_signature_match?(normalized_body, signature) }
end