Class: ContextDev::Resources::Web

Inherits:
Object
  • Object
show all
Defined in:
lib/context_dev/resources/web.rb,
sig/context_dev/resources/web.rbs

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ Web

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Web.

Parameters:



620
621
622
 
# File 'lib/context_dev/resources/web.rb', line 620

def initialize(client:)
  @client = client
end

Instance Method Details

#extract(schema:, url:, actions: nil, fact_check: nil, follow_subdomains: nil, include_frames: nil, instructions: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractResponse

Some parameter documentations has been truncated, see Models::WebExtractParams for more details.

Crawl a website, use the provided JSON Schema and instructions to prioritize relevant internal links, and extract structured data from the selected pages.

Parameters:

  • schema (Hash{Symbol=>Object}) —

    JSON Schema for the returned data object. Image fields such as image_urls or `

  • url (String) —

    The starting website URL to crawl and extract from. Must include http:// or http

  • actions (Array<ContextDev::Models::WebExtractParams::Action::Wait, ContextDev::Models::WebExtractParams::Action::Perform, ContextDev::Models::WebExtractParams::Action::Scroll>) —

    Optional browser actions executed in order on the requested page after it loads,

  • fact_check (Boolean) —

    When true, every returned value must be grounded in facts stated on the page; fi

  • follow_subdomains (Boolean) —

    When true, follow links on subdomains of the starting URL's domain.

  • include_frames (Boolean) —

    When true, iframe contents are included in Markdown before extraction.

  • instructions (String) —

    Optional extraction guidance, such as which facts to prioritize or how to interp

  • max_age_ms (Integer) —

    Return cached scrape results if a prior scrape for the same parameters is younge

  • max_depth (Integer) —

    Optional maximum link depth from the starting URL (0 = only the starting page).

  • max_pages (Integer) —

    Maximum number of pages to analyze for extraction. Hard cap: 50. Defaults to 5.

  • pdf (ContextDev::Models::WebExtractParams::Pdf)
  • settle_animations (Boolean) —

    When true, waits briefly for CSS and transition animations to settle before extr

  • stop_after_ms (Integer) —

    Soft time budget for the crawl in milliseconds. Min: 10000 (10s). Max: 110000 (1

  • tags (Array<String>) —

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • wait_for_ms (Integer) —

    Optional browser wait time in milliseconds after initial page load for each craw

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



51
52
53
54
55
56
57
58
59
60
 
# File 'lib/context_dev/resources/web.rb', line 51

def extract(params)
  parsed, options = ContextDev::WebExtractParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/extract",
    body: parsed,
    model: ContextDev::Models::WebExtractResponse,
    options: options
  )
end

#extract_competitors(domain:, num_competitors: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractCompetitorsResponse

Some parameter documentations has been truncated, see Models::WebExtractCompetitorsParams for more details.

Analyze a company's landing page and web search evidence to return direct competitors for the same product or market.

Parameters:

  • domain (String) —

    Company domain to analyze, such as stripe.com. Full http(s) URLs are accepted

  • num_competitors (Integer) —

    Exact number of direct competitors to return. Defaults to 5.

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/context_dev/resources/web.rb', line 83

def extract_competitors(params)
  parsed, options = ContextDev::WebExtractCompetitorsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/competitors",
    query: query.transform_keys(num_competitors: "numCompetitors", timeout_ms: "timeoutMS"),
    model: ContextDev::Models::WebExtractCompetitorsResponse,
    options: options
  )
end

#extract_fonts(direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractFontsResponse

Some parameter documentations has been truncated, see Models::WebExtractFontsParams for more details.

Scrape font information from a website including font families, usage statistics, fallbacks, and element/word counts.

Parameters:

  • direct_url (String) —

    A specific URL to fetch fonts from directly, bypassing domain resolution (e.g.,

  • domain (String) —

    Domain name to extract fonts from (e.g., 'example.com', 'google.com'). The domai

  • max_age_ms (Integer, nil) —

    Maximum age in milliseconds for cached brand data before the API performs a hard

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/context_dev/resources/web.rb', line 118

def extract_fonts(params = {})
  parsed, options = ContextDev::WebExtractFontsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/fonts",
    query: query.transform_keys(
      direct_url: "directUrl",
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS"
    ),
    model: ContextDev::Models::WebExtractFontsResponse,
    options: options
  )
end

#extract_styleguide(color_scheme: nil, direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractStyleguideResponse

Some parameter documentations has been truncated, see Models::WebExtractStyleguideParams for more details.

Extract a comprehensive design system from a website including colors, typography, spacing, shadows, and UI components.

Parameters:

  • color_scheme (Symbol, ContextDev::Models::WebExtractStyleguideParams::ColorScheme) —

    Optional browser color scheme to emulate for websites that respond to prefers-co

  • direct_url (String) —

    A specific URL to fetch the styleguide from directly, bypassing domain resolutio

  • domain (String) —

    Domain name to extract styleguide from (e.g., 'example.com', 'google.com'). The

  • max_age_ms (Integer, nil) —

    Maximum age in milliseconds for cached brand data before the API performs a hard

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
 
# File 'lib/context_dev/resources/web.rb', line 159

def extract_styleguide(params = {})
  parsed, options = ContextDev::WebExtractStyleguideParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/styleguide",
    query: query.transform_keys(
      color_scheme: "colorScheme",
      direct_url: "directUrl",
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS"
    ),
    model: ContextDev::Models::WebExtractStyleguideResponse,
    options: options
  )
end

#screenshot(clear_popups: nil, color_scheme: nil, country: nil, direct_url: nil, domain: nil, full_screenshot: nil, handle_cookie_popup: nil, max_age_ms: nil, page: nil, scroll_offset: nil, tags: nil, timeout_ms: nil, viewport: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebScreenshotResponse

Some parameter documentations has been truncated, see Models::WebScreenshotParams for more details.

Capture a screenshot of a website.

Parameters:

  • clear_popups (Boolean) —

    Optional parameter for comprehensive popup cleanup. If 'true', the browser dismi

  • color_scheme (Symbol, ContextDev::Models::WebScreenshotParams::ColorScheme) —

    Optional parameter to choose the site's visual theme in the screenshot. Use 'lig

  • country (Symbol, ContextDev::Models::WebScreenshotParams::Country) —

    Fetch the target page through a residential proxy in this country (ISO 3166-1 al

  • direct_url (String) —

    A specific URL to screenshot directly, bypassing domain resolution (e.g., 'https

  • domain (String) —

    Domain name to take screenshot of (e.g., 'example.com', 'google.com'). The domai

  • full_screenshot (Symbol, ContextDev::Models::WebScreenshotParams::FullScreenshot) —

    Optional parameter to determine screenshot type. If 'true', takes a full page sc

  • handle_cookie_popup (Boolean) —

    Optional parameter to control cookie/consent popup handling. If 'true', we dismi

  • max_age_ms (Integer, nil) —

    Return a cached screenshot if a prior screenshot for the same parameters exists

  • page (Symbol, ContextDev::Models::WebScreenshotParams::Page) —

    Optional parameter to specify which page type to screenshot. If provided, the sy

  • scroll_offset (Integer, nil) —

    Optional vertical scroll offset in pixels for capturing a long page in viewport-

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • viewport (ContextDev::Models::WebScreenshotParams::Viewport) —

    Optional browser viewport dimensions for the screenshot. Defaults to 1920x1080.

  • wait_for_ms (Integer, nil) —

    Optional browser wait time in milliseconds after initial page load before taking

  • zdr (Symbol, ContextDev::Models::WebScreenshotParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
 
# File 'lib/context_dev/resources/web.rb', line 218

def screenshot(params = {})
  parsed, options = ContextDev::WebScreenshotParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/screenshot",
    query: query.transform_keys(
      clear_popups: "clearPopups",
      color_scheme: "colorScheme",
      direct_url: "directUrl",
      full_screenshot: "fullScreenshot",
      handle_cookie_popup: "handleCookiePopup",
      max_age_ms: "maxAgeMs",
      scroll_offset: "scrollOffset",
      timeout_ms: "timeoutMS",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebScreenshotResponse,
    options: options
  )
end

#search(query:, country: nil, exclude_domains: nil, freshness: nil, include_domains: nil, markdown_options: nil, num_results: nil, query_fanout: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebSearchResponse

Some parameter documentations has been truncated, see Models::WebSearchParams for more details.

Search the web and optionally scrape each result to Markdown in one round-trip.

Parameters:

  • query (String) —

    Search query. Accepts natural language as well as Google-style search operators

  • country (Symbol, ContextDev::Models::WebSearchParams::Country) —

    Two-letter ISO 3166-1 alpha-2 country code to localize results to a specific cou

  • exclude_domains (Array<String>) —

    Blocklist — drop results from these domains. Example: ["pinterest.com", "reddit.

  • freshness (Symbol, ContextDev::Models::WebSearchParams::Freshness) —

    Restrict results to content published within this window.

  • include_domains (Array<String>) —

    Allowlist — only return results from these domains. Example: ["arxiv.org", "gith

  • markdown_options (ContextDev::Models::WebSearchParams::MarkdownOptions) —

    Inline Markdown scraping for each result. Set enabled: true to activate.

  • num_results (Integer) —

    Number of results to request and return (10–100). Defaults to 10.

  • query_fanout (Boolean) —

    Expand the query into multiple parallel variants for broader recall.

  • tags (Array<String>) —

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



272
273
274
275
276
277
278
279
280
281
 
# File 'lib/context_dev/resources/web.rb', line 272

def search(params)
  parsed, options = ContextDev::WebSearchParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/search",
    body: parsed,
    model: ContextDev::Models::WebSearchResponse,
    options: options
  )
end

#web_crawl_md(url:, country: nil, exclude_selectors: nil, follow_subdomains: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, url_regex: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebCrawlMdResponse

Some parameter documentations has been truncated, see Models::WebWebCrawlMdParams for more details.

Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages.

Parameters:

  • url (String) —

    The starting URL for the crawl (must include http:// or https:// protocol)

  • country (Symbol, ContextDev::Models::WebWebCrawlMdParams::Country) —

    Fetch the target page through a residential proxy in this country (ISO 3166-1 al

  • exclude_selectors (Array<String>) —

    CSS selectors to remove before each crawled page is converted to Markdown. Appli

  • follow_subdomains (Boolean) —

    When true, follow links on subdomains of the starting URL's domain (e.g. docs.ex

  • include_frames (Boolean) —

    When true, the contents of iframes are rendered to Markdown for each crawled pag

  • include_images (Boolean) —

    Include image references in the Markdown output

  • include_links (Boolean) —

    Preserve hyperlinks in the Markdown output

  • include_selectors (Array<String>) —

    CSS selectors. When provided, only matching HTML subtrees (and their descendants

  • max_age_ms (Integer) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • max_depth (Integer) —

    Maximum link depth from the starting URL (0 = only the starting page)

  • max_pages (Integer) —

    Maximum number of pages to crawl. Hard cap: 500.

  • pdf (ContextDev::Models::WebWebCrawlMdParams::Pdf) —

    PDF parsing controls. Use start/end to limit text extraction and embedded-image

  • settle_animations (Boolean) —

    When true, waits briefly for CSS and transition animations to settle before extr

  • shorten_base64_images (Boolean) —

    Truncate base64-encoded image data in the Markdown output

  • stop_after_ms (Integer) —

    Soft time budget for the crawl in milliseconds. After each scrape, the crawler c

  • tags (Array<String>) —

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • url_regex (String) —

    Regex pattern. Only URLs matching this pattern will be followed and scraped. An

  • use_main_content_only (Boolean) —

    Extract only the main content, stripping headers, footers, sidebars, and navigat

  • wait_for_ms (Integer) —

    Browser wait time in milliseconds after initial page load for each crawled page.

  • zdr (Symbol, ContextDev::Models::WebWebCrawlMdParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



338
339
340
341
342
343
344
345
346
347
 
# File 'lib/context_dev/resources/web.rb', line 338

def web_crawl_md(params)
  parsed, options = ContextDev::WebWebCrawlMdParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/crawl",
    body: parsed,
    model: ContextDev::Models::WebWebCrawlMdResponse,
    options: options
  )
end

#web_scrape_html(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeHTMLResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeHTMLParams for more details.

Scrapes the given URL and returns the raw HTML content of the page. The base request costs 1 credit; requests with browser actions cost 2 credits.

Parameters:

  • url (String) —

    Full URL to scrape (must include http:// or https:// protocol)

  • actions (Array<ContextDev::Models::WebWebScrapeHTMLParams::Action::Wait, ContextDev::Models::WebWebScrapeHTMLParams::Action::Perform, ContextDev::Models::WebWebScrapeHTMLParams::Action::Scroll>, nil) —

    Optional browser actions executed in array order after the page loads and before

  • country (Symbol, ContextDev::Models::WebWebScrapeHTMLParams::Country) —

    Fetch the target page through a residential proxy in this country (ISO 3166-1 al

  • exclude_selectors (Array<String>, nil) —

    CSS selectors to remove from the result. Applied after includeSelectors. Exclusi

  • headers (Hash{Symbol=>String}) —

    Optional outbound HTTP headers forwarded only to the target URL, sent as deep-ob

  • include_frames (Boolean) —

    When true, iframes are rendered inline into the returned HTML.

  • include_selectors (Array<String>, nil) —

    CSS selectors. When provided, only matching subtrees (and their descendants) are

  • max_age_ms (Integer, nil) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • pdf (ContextDev::Models::WebWebScrapeHTMLParams::Pdf) —

    PDF parsing controls. Use start/end to limit text extraction and embedded-image

  • settle_animations (Boolean) —

    When true, waits briefly for CSS and transition animations to settle before extr

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • use_main_content_only (Boolean) —

    When true, return only the page's main content in the HTML response, excluding h

  • wait_for_ms (Integer, nil) —

    Optional browser wait time in milliseconds after initial page load. Min: 0. Max:

  • zdr (Symbol, ContextDev::Models::WebWebScrapeHTMLParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
 
# File 'lib/context_dev/resources/web.rb', line 392

def web_scrape_html(params)
  parsed, options = ContextDev::WebWebScrapeHTMLParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/html",
    query: query.transform_keys(
      exclude_selectors: "excludeSelectors",
      include_frames: "includeFrames",
      include_selectors: "includeSelectors",
      max_age_ms: "maxAgeMs",
      settle_animations: "settleAnimations",
      timeout_ms: "timeoutMS",
      use_main_content_only: "useMainContentOnly",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeHTMLResponse,
    options: options
  )
end

#web_scrape_images(url:, actions: nil, dedupe: nil, enrichment: nil, headers: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeImagesResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeImagesParams for more details.

Extract image assets from a web page, including standard URLs, inline SVGs, data URIs, responsive image sources, metadata, CSS backgrounds, video posters, and embeds. The base request costs 1 credit, or 2 credits with browser actions. When enrichment is enabled, the entire call costs 5 credits, including requests that also use actions.

Parameters:

Returns:

See Also:



447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
 
# File 'lib/context_dev/resources/web.rb', line 447

def web_scrape_images(params)
  parsed, options = ContextDev::WebWebScrapeImagesParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/images",
    query: query.transform_keys(
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeImagesResponse,
    options: options
  )
end

#web_scrape_md(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_html: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeMdResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeMdParams for more details.

Scrapes the given URL into LLM usable Markdown. Inspect key_metadata on JSON responses from a recognized API key; use error_code to distinguish stable failure categories.

YouTube

YouTube URLs return the video or channel itself rather than the surrounding player and navigation chrome. A URL addressing a single video (/watch, youtu.be, /shorts, /embed, /live) returns its title, channel, duration, view count, keywords, full description, and the transcript when the video has captions that can be retrieved; videos without captions return everything except the transcript. A channel URL (/channel/UC…, /@handle, /c/…, /user/…) returns its name, handle, subscriber count, video count, and full description. When includeImages=true, video responses also include the thumbnail and channel responses include the avatar. Costs the same as any other scrape.

Billing & errors

HTTP status Billed? Meaning
200 Yes — 1 credit, or 2 credits with actions Successful scrape, including a zero-length result when includeSelectors matched nothing
400 No Invalid input, skipped PDF, or the page could not be scraped. error_code WEBSITE_BLOCKED specifically means the site answered with an anti-bot challenge, CAPTCHA wall, or login shell instead of the page (even when the site returned HTTP 200) — retrying later or from another country sometimes succeeds
401 / 403 No Invalid/disabled key, insufficient permissions, or credits exhausted; inspect error_code
404 No Target page returned or fingerprinted as not found
408 No Request timed out
413 No Target content exceeds the maximum supported size (20 MB)
415 No Unsupported content type
429 No Per-minute rate limit exceeded; honor Retry-After
500 No Internal error

Parameters:

  • url (String) —

    Full URL to scrape into LLM usable Markdown (must include http:// or https:// pr

  • actions (Array<ContextDev::Models::WebWebScrapeMdParams::Action::Wait, ContextDev::Models::WebWebScrapeMdParams::Action::Perform, ContextDev::Models::WebWebScrapeMdParams::Action::Scroll>, nil) —

    Optional browser actions executed in array order after the page loads and before

  • country (Symbol, ContextDev::Models::WebWebScrapeMdParams::Country) —

    Fetch the target page through a residential proxy in this country (ISO 3166-1 al

  • exclude_selectors (Array<String>, nil) —

    CSS selectors to remove before conversion to Markdown. Applied after includeSele

  • headers (Hash{Symbol=>String}) —

    Optional outbound HTTP headers forwarded only to the target URL, sent as deep-ob

  • include_frames (Boolean) —

    When true, the contents of iframes are rendered to Markdown.

  • include_html (Boolean) —

    When true, the response also includes an html field with the page HTML the Mar

  • include_images (Boolean) —

    Include image references in Markdown output

  • include_links (Boolean) —

    Preserve hyperlinks in Markdown output

  • include_selectors (Array<String>, nil) —

    CSS selectors. When provided, only matching HTML subtrees (and their descendants

  • max_age_ms (Integer, nil) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • pdf (ContextDev::Models::WebWebScrapeMdParams::Pdf) —

    PDF parsing controls. Use start/end to limit text extraction and embedded-image

  • settle_animations (Boolean) —

    When true, waits briefly for CSS and transition animations to settle before conv

  • shorten_base64_images (Boolean) —

    Shorten base64-encoded image data in the Markdown output

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • use_main_content_only (Boolean) —

    Extract only the main content of the page, excluding headers, footers, sidebars,

  • wait_for_ms (Integer, nil) —

    Optional browser wait time in milliseconds after initial page load before conver

  • zdr (Symbol, ContextDev::Models::WebWebScrapeMdParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
 
# File 'lib/context_dev/resources/web.rb', line 541

def web_scrape_md(params)
  parsed, options = ContextDev::WebWebScrapeMdParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/markdown",
    query: query.transform_keys(
      exclude_selectors: "excludeSelectors",
      include_frames: "includeFrames",
      include_html: "includeHTML",
      include_images: "includeImages",
      include_links: "includeLinks",
      include_selectors: "includeSelectors",
      max_age_ms: "maxAgeMs",
      settle_animations: "settleAnimations",
      shorten_base64_images: "shortenBase64Images",
      timeout_ms: "timeoutMS",
      use_main_content_only: "useMainContentOnly",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeMdResponse,
    options: options
  )
end

#web_scrape_sitemap(domain:, headers: nil, max_links: nil, search: nil, sitemap_url: nil, tags: nil, timeout_ms: nil, url_regex: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeSitemapResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeSitemapParams for more details.

Crawl an entire website's sitemap and return all discovered page URLs. Pass search to have the crawled sitemap filtered down to the pages about a phrase (for example pricing and plans or api authentication docs), most relevant first — a searched crawl scans the whole sitemap and costs 2 credits instead of 1.

Parameters:

  • domain (String) —

    Domain to build a sitemap for

  • headers (Hash{Symbol=>String}) —

    Optional outbound HTTP headers forwarded only to the target URL, sent as deep-ob

  • max_links (Integer) —

    Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Mi

  • search (String) —

    Optional search phrase. When provided, the crawled sitemap is filtered to the pa

  • sitemap_url (String) —

    Optional explicit sitemap URL. When provided, exactly this sitemap is crawled in

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • url_regex (String) —

    Optional RE2-compatible regex pattern. Only URLs matching this pattern are retur

  • zdr (Symbol, ContextDev::Models::WebWebScrapeSitemapParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
 
# File 'lib/context_dev/resources/web.rb', line 600

def web_scrape_sitemap(params)
  parsed, options = ContextDev::WebWebScrapeSitemapParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/sitemap",
    query: query.transform_keys(
      max_links: "maxLinks",
      sitemap_url: "sitemapUrl",
      timeout_ms: "timeoutMS",
      url_regex: "urlRegex"
    ),
    model: ContextDev::Models::WebWebScrapeSitemapResponse,
    options: options
  )
end