Module: Html2rss::MCP::Inspect

Defined in:
lib/html2rss/mcp/inspect.rb

Overview

Diagnostic inspect path (not Capture ownership). Fetches once, then delegates shared recon to PageRecon; adds MCP-only scraper/XHR diagnostics.

Class Method Summary collapse

Class Method Details

.call(url:, strategy: :auto) ⇒ Hash

Parameters:

  • url (String)
  • strategy (String, Symbol) (defaults to: :auto)

Returns:

  • (Hash) 


15
16
17
18
19
20
21
22
23
24
25
26
 
# File 'lib/html2rss/mcp/inspect.rb', line 15

def call(url:, strategy: :auto)
  resolved = FeedPipeline::StrategyPlan.concrete_for_diagnostic(strategy)
  response = fetch_response(url, resolved)
  recon = PageRecon.call(response:, url:, strategy: resolved)

  result = recon.to_h.merge(
    strategy: resolved,
    scraper_eligibility: scraper_info(safe_parsed_body(response))
  )
  result[:xhr_capture] = xhr_capture_info(response) if resolved == :botasaurus
  result
end

.discover_segments(sst, url) ⇒ Array

Parameters:

Returns:

  • (Array) 


124
 
# File 'lib/html2rss/mcp/inspect.rb', line 124

def discover_segments(sst, url) = PageRecon.discover_segments(sst, url)

.fetch_response(url, strategy) ⇒ Html2rss::RequestService::Response

Parameters:

  • url (String)
  • strategy (Symbol)

Returns:



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/html2rss/mcp/inspect.rb', line 75

def fetch_response(url, strategy) # rubocop:disable Metrics/MethodLength -- session construction
  raw_config = Config.auto_source_config(
    url:,
    request_controls: Config::RequestControls.from_shortcut(strategy:)
  )
  raw_config[:strategy] = strategy
  config = Config.from_hash(raw_config)
  resources = FeedPipeline::RuntimePolicy.resources_for(config)
  session = RequestSession.build(
    config:,
    strategy: config.strategy,
    budget: resources.budget,
    policy: resources.policy
  )
  session.fetch_initial_response
end

.redacted_endpoint(entry) ⇒ String?

Returns scheme+host+path only.

Parameters:

  • entry (Hash) —

    captured response hash

Returns:

  • (String, nil) —

    scheme+host+path only



44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/html2rss/mcp/inspect.rb', line 44

def redacted_endpoint(entry)
  raw = entry['url'] || entry[:url]
  return unless raw

  uri = URI.parse(raw.to_s)
  return unless uri.scheme && uri.host

  "#{uri.scheme}://#{uri.host}#{uri.path}"
rescue URI::InvalidURIError
  nil
end

.scraper_info(parsed) ⇒ Array<String>, Hash

Parameters:

  • parsed (Object) —

    parsed response body

Returns:

  • (Array<String>, Hash) 


96
97
98
99
100
101
102
103
104
 
# File 'lib/html2rss/mcp/inspect.rb', line 96

def scraper_info(parsed)
  return { error: 'Response is not HTML' } unless parsed.is_a?(Nokogiri::HTML::Document)

  begin
    Html2rss::AutoSource::Scraper.from(parsed).map(&:name)
  rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
    { none_found: error.category.to_s }
  end
end

.sst_stats_from(response) ⇒ Hash?

Parameters:

Returns:

  • (Hash, nil) 


110
111
112
113
114
115
116
117
 
# File 'lib/html2rss/mcp/inspect.rb', line 110

def sst_stats_from(response)
  return nil unless response.html_response?

  recon = PageRecon.call(response:, url: response.url)
  return nil unless recon.sst

  { node_count: recon.sst[:node_count], degraded: recon.sst[:degraded] }
end

.xhr_candidate_articles?(entry) ⇒ Boolean

Parameters:

  • entry (Hash) —

    captured response hash

Returns:

  • (Boolean) 


60
61
62
63
64
65
66
67
68
 
# File 'lib/html2rss/mcp/inspect.rb', line 60

def xhr_candidate_articles?(entry)
  body = entry['body'] || entry[:body]
  return false unless body.is_a?(String)

  document = JSON.parse(body, symbolize_names: true)
  AutoSource::Scraper::JsonState::CandidateDetector.candidate_array?(document)
rescue JSON::ParserError
  false
end

.xhr_capture_info(response) ⇒ Hash

Returns redacted XHR capture diagnostics (no query strings).

Parameters:

Returns:

  • (Hash) —

    redacted XHR capture diagnostics (no query strings)



31
32
33
34
35
36
37
38
 
# File 'lib/html2rss/mcp/inspect.rb', line 31

def xhr_capture_info(response)
  captured = response.captured_responses
  {
    count: captured.size,
    sample_endpoints: captured.first(5).filter_map { |entry| redacted_endpoint(entry) },
    candidate_articles: captured.any? { |entry| xhr_candidate_articles?(entry) }
  }
end