Class: ContextDev::Resources::Web

Inherits:
Object
  • Object
show all
Defined in:
lib/context_dev/resources/web.rb,
sig/context_dev/resources/web.rbs

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ Web

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Web.

Parameters:



612
613
614
# File 'lib/context_dev/resources/web.rb', line 612

def initialize(client:)
  @client = client
end

Instance Method Details

#extract(schema:, url:, fact_check: nil, follow_subdomains: nil, include_frames: nil, instructions: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractResponse

Some parameter documentations has been truncated, see Models::WebExtractParams for more details.

Crawl a website, use the provided JSON Schema and instructions to prioritize relevant internal links, and extract structured data from the selected pages.

Parameters:

  • schema (Hash{Symbol=>Object})

    JSON Schema for the returned data object. TypeScript Zod users can pass a JSON S

  • url (String)

    The starting website URL to crawl and extract from. Must include http:// or http

  • fact_check (Boolean)

    When true, every returned value must be grounded in facts stated on the page; fi

  • follow_subdomains (Boolean)

    When true, follow links on subdomains of the starting URL's domain.

  • include_frames (Boolean)

    When true, iframe contents are included in Markdown before extraction.

  • instructions (String)

    Optional extraction guidance, such as which facts to prioritize or how to interp

  • max_age_ms (Integer)

    Return cached scrape results if a prior scrape for the same parameters is younge

  • max_depth (Integer)

    Optional maximum link depth from the starting URL (0 = only the starting page).

  • max_pages (Integer)

    Maximum number of pages to analyze for extraction. Hard cap: 50. Defaults to 5.

  • pdf (ContextDev::Models::WebExtractParams::Pdf)
  • settle_animations (Boolean)

    When true, waits briefly for CSS and transition animations to settle before extr

  • stop_after_ms (Integer)

    Soft time budget for the crawl in milliseconds. Min: 10000 (10s). Max: 110000 (1

  • tags (Array<String>)

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • wait_for_ms (Integer)

    Optional browser wait time in milliseconds after initial page load for each craw

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



49
50
51
52
53
54
55
56
57
58
# File 'lib/context_dev/resources/web.rb', line 49

def extract(params)
  parsed, options = ContextDev::WebExtractParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/extract",
    body: parsed,
    model: ContextDev::Models::WebExtractResponse,
    options: options
  )
end

#extract_competitors(domain:, num_competitors: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractCompetitorsResponse

Some parameter documentations has been truncated, see Models::WebExtractCompetitorsParams for more details.

Analyze a company's landing page and web search evidence to return direct competitors for the same product or market.

Parameters:

  • domain (String)

    Company domain to analyze, such as stripe.com. Full http(s) URLs are accepted

  • num_competitors (Integer)

    Exact number of direct competitors to return. Defaults to 5.

  • tags (Array<String>)

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



81
82
83
84
85
86
87
88
89
90
91
# File 'lib/context_dev/resources/web.rb', line 81

def extract_competitors(params)
  parsed, options = ContextDev::WebExtractCompetitorsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/competitors",
    query: query.transform_keys(num_competitors: "numCompetitors", timeout_ms: "timeoutMS"),
    model: ContextDev::Models::WebExtractCompetitorsResponse,
    options: options
  )
end

#extract_fonts(direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractFontsResponse

Some parameter documentations has been truncated, see Models::WebExtractFontsParams for more details.

Scrape font information from a website including font families, usage statistics, fallbacks, and element/word counts.

Parameters:

  • direct_url (String)

    A specific URL to fetch fonts from directly, bypassing domain resolution (e.g.,

  • domain (String)

    Domain name to extract fonts from (e.g., 'example.com', 'google.com'). The domai

  • max_age_ms (Integer, nil)

    Maximum age in milliseconds for cached brand data before the API performs a hard

  • tags (Array<String>)

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
# File 'lib/context_dev/resources/web.rb', line 116

def extract_fonts(params = {})
  parsed, options = ContextDev::WebExtractFontsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/fonts",
    query: query.transform_keys(
      direct_url: "directUrl",
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS"
    ),
    model: ContextDev::Models::WebExtractFontsResponse,
    options: options
  )
end

#extract_styleguide(color_scheme: nil, direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractStyleguideResponse

Some parameter documentations has been truncated, see Models::WebExtractStyleguideParams for more details.

Extract a comprehensive design system from a website including colors, typography, spacing, shadows, and UI components.

Parameters:

  • color_scheme (Symbol, ContextDev::Models::WebExtractStyleguideParams::ColorScheme)

    Optional browser color scheme to emulate for websites that respond to prefers-co

  • direct_url (String)

    A specific URL to fetch the styleguide from directly, bypassing domain resolutio

  • domain (String)

    Domain name to extract styleguide from (e.g., 'example.com', 'google.com'). The

  • max_age_ms (Integer, nil)

    Maximum age in milliseconds for cached brand data before the API performs a hard

  • tags (Array<String>)

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
# File 'lib/context_dev/resources/web.rb', line 157

def extract_styleguide(params = {})
  parsed, options = ContextDev::WebExtractStyleguideParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/styleguide",
    query: query.transform_keys(
      color_scheme: "colorScheme",
      direct_url: "directUrl",
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS"
    ),
    model: ContextDev::Models::WebExtractStyleguideResponse,
    options: options
  )
end

#screenshot(color_scheme: nil, country: nil, direct_url: nil, domain: nil, full_screenshot: nil, handle_cookie_popup: nil, max_age_ms: nil, page: nil, scroll_offset: nil, tags: nil, timeout_ms: nil, viewport: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebScreenshotResponse

Some parameter documentations has been truncated, see Models::WebScreenshotParams for more details.

Capture a screenshot of a website.

Parameters:

  • color_scheme (Symbol, ContextDev::Models::WebScreenshotParams::ColorScheme)

    Optional parameter to choose the site's visual theme in the screenshot. Use 'lig

  • country (Symbol, ContextDev::Models::WebScreenshotParams::Country)

    Fetch the target page through a residential proxy in this country (ISO 3166-1 al

  • direct_url (String)

    A specific URL to screenshot directly, bypassing domain resolution (e.g., 'https

  • domain (String)

    Domain name to take screenshot of (e.g., 'example.com', 'google.com'). The domai

  • full_screenshot (Symbol, ContextDev::Models::WebScreenshotParams::FullScreenshot)

    Optional parameter to determine screenshot type. If 'true', takes a full page sc

  • handle_cookie_popup (Boolean, Symbol, ContextDev::Models::WebScreenshotParams::HandleCookiePopup)

    Optional parameter to control cookie/consent popup handling. If 'true', we dismi

  • max_age_ms (Integer, nil)

    Return a cached screenshot if a prior screenshot for the same parameters exists

  • page (Symbol, ContextDev::Models::WebScreenshotParams::Page)

    Optional parameter to specify which page type to screenshot. If provided, the sy

  • scroll_offset (Integer, nil)

    Optional vertical scroll offset in pixels for capturing a long page in viewport-

  • tags (Array<String>)

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • viewport (ContextDev::Models::WebScreenshotParams::Viewport)

    Optional browser viewport dimensions for the screenshot. Defaults to 1920x1080.

  • wait_for_ms (Integer, nil)

    Optional browser wait time in milliseconds after initial page load before taking

  • zdr (Symbol, ContextDev::Models::WebScreenshotParams::Zdr)

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
# File 'lib/context_dev/resources/web.rb', line 214

def screenshot(params = {})
  parsed, options = ContextDev::WebScreenshotParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/screenshot",
    query: query.transform_keys(
      color_scheme: "colorScheme",
      direct_url: "directUrl",
      full_screenshot: "fullScreenshot",
      handle_cookie_popup: "handleCookiePopup",
      max_age_ms: "maxAgeMs",
      scroll_offset: "scrollOffset",
      timeout_ms: "timeoutMS",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebScreenshotResponse,
    options: options
  )
end

#search(query:, country: nil, exclude_domains: nil, freshness: nil, include_domains: nil, markdown_options: nil, num_results: nil, query_fanout: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebSearchResponse

Some parameter documentations has been truncated, see Models::WebSearchParams for more details.

Search the web and optionally scrape each result to Markdown in one round-trip.

Parameters:

  • query (String)

    Search query. Accepts natural language as well as Google-style search operators

  • country (Symbol, ContextDev::Models::WebSearchParams::Country)

    Two-letter ISO 3166-1 alpha-2 country code to localize results to a specific cou

  • exclude_domains (Array<String>)

    Blocklist — drop results from these domains. Example: ["pinterest.com", "reddit.

  • freshness (Symbol, ContextDev::Models::WebSearchParams::Freshness)

    Restrict results to content published within this window.

  • include_domains (Array<String>)

    Allowlist — only return results from these domains. Example: ["arxiv.org", "gith

  • markdown_options (ContextDev::Models::WebSearchParams::MarkdownOptions)

    Inline Markdown scraping for each result. Set enabled: true to activate.

  • num_results (Integer)

    Number of results to request and return (10–100). Defaults to 10.

  • query_fanout (Boolean)

    Expand the query into multiple parallel variants for broader recall.

  • tags (Array<String>)

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



267
268
269
270
271
272
273
274
275
276
# File 'lib/context_dev/resources/web.rb', line 267

def search(params)
  parsed, options = ContextDev::WebSearchParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/search",
    body: parsed,
    model: ContextDev::Models::WebSearchResponse,
    options: options
  )
end

#web_crawl_md(url:, country: nil, exclude_selectors: nil, follow_subdomains: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, url_regex: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebCrawlMdResponse

Some parameter documentations has been truncated, see Models::WebWebCrawlMdParams for more details.

Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages.

Parameters:

  • url (String)

    The starting URL for the crawl (must include http:// or https:// protocol)

  • country (Symbol, ContextDev::Models::WebWebCrawlMdParams::Country)

    Fetch the target page through a residential proxy in this country (ISO 3166-1 al

  • exclude_selectors (Array<String>)

    CSS selectors to remove before each crawled page is converted to Markdown. Appli

  • follow_subdomains (Boolean)

    When true, follow links on subdomains of the starting URL's domain (e.g. docs.ex

  • include_frames (Boolean)

    When true, the contents of iframes are rendered to Markdown for each crawled pag

  • include_images (Boolean)

    Include image references in the Markdown output

  • include_links (Boolean)

    Preserve hyperlinks in the Markdown output

  • include_selectors (Array<String>)

    CSS selectors. When provided, only matching HTML subtrees (and their descendants

  • max_age_ms (Integer)

    Return a cached result if a prior scrape for the same parameters exists and is y

  • max_depth (Integer)

    Maximum link depth from the starting URL (0 = only the starting page)

  • max_pages (Integer)

    Maximum number of pages to crawl. Hard cap: 500.

  • pdf (ContextDev::Models::WebWebCrawlMdParams::Pdf)

    PDF parsing controls. Use start/end to limit text extraction and embedded-image

  • settle_animations (Boolean)

    When true, waits briefly for CSS and transition animations to settle before extr

  • shorten_base64_images (Boolean)

    Truncate base64-encoded image data in the Markdown output

  • stop_after_ms (Integer)

    Soft time budget for the crawl in milliseconds. After each scrape, the crawler c

  • tags (Array<String>)

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • url_regex (String)

    Regex pattern. Only URLs matching this pattern will be followed and scraped.

  • use_main_content_only (Boolean)

    Extract only the main content, stripping headers, footers, sidebars, and navigat

  • wait_for_ms (Integer)

    Optional browser wait time in milliseconds after initial page load for each craw

  • zdr (Symbol, ContextDev::Models::WebWebCrawlMdParams::Zdr)

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



333
334
335
336
337
338
339
340
341
342
# File 'lib/context_dev/resources/web.rb', line 333

def web_crawl_md(params)
  parsed, options = ContextDev::WebWebCrawlMdParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/crawl",
    body: parsed,
    model: ContextDev::Models::WebWebCrawlMdResponse,
    options: options
  )
end

#web_scrape_html(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeHTMLResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeHTMLParams for more details.

Scrapes the given URL and returns the raw HTML content of the page. The base request costs 1 credit; requests with browser actions cost 2 credits.

Parameters:

Returns:

See Also:



387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
# File 'lib/context_dev/resources/web.rb', line 387

def web_scrape_html(params)
  parsed, options = ContextDev::WebWebScrapeHTMLParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/html",
    query: query.transform_keys(
      exclude_selectors: "excludeSelectors",
      include_frames: "includeFrames",
      include_selectors: "includeSelectors",
      max_age_ms: "maxAgeMs",
      settle_animations: "settleAnimations",
      timeout_ms: "timeoutMS",
      use_main_content_only: "useMainContentOnly",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeHTMLResponse,
    options: options
  )
end

#web_scrape_images(url:, actions: nil, dedupe: nil, enrichment: nil, headers: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeImagesResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeImagesParams for more details.

Extract image assets from a web page, including standard URLs, inline SVGs, data URIs, responsive image sources, metadata, CSS backgrounds, video posters, and embeds. The base request costs 1 credit, or 2 credits with browser actions. When enrichment is enabled, the entire call costs 5 credits, including requests that also use actions.

Parameters:

Returns:

See Also:



442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
# File 'lib/context_dev/resources/web.rb', line 442

def web_scrape_images(params)
  parsed, options = ContextDev::WebWebScrapeImagesParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/images",
    query: query.transform_keys(
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeImagesResponse,
    options: options
  )
end

#web_scrape_md(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeMdResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeMdParams for more details.

Scrapes the given URL into LLM usable Markdown. Inspect key_metadata on JSON responses from a recognized API key; use error_code to distinguish stable failure categories.

YouTube

YouTube URLs return the video or channel itself rather than the surrounding player and navigation chrome. A URL addressing a single video (/watch, youtu.be, /shorts, /embed, /live) returns its title, channel, duration, view count, keywords, full description, and the transcript when the video has captions that can be retrieved; videos without captions return everything except the transcript. A channel URL (/channel/UC…, /@handle, /c/…, /user/…) returns its name, handle, subscriber count, video count, and full description. When includeImages=true, video responses also include the thumbnail and channel responses include the avatar. Costs the same as any other scrape.

Billing & errors

HTTP status Billed? Meaning
200 Yes — 1 credit, or 2 credits with actions Successful scrape, including a zero-length result when includeSelectors matched nothing
400 No Invalid input, skipped PDF, or the page could not be scraped
401 / 403 No Invalid/disabled key, insufficient permissions, or credits exhausted; inspect error_code
404 No Target page returned or fingerprinted as not found
408 No Request timed out
413 No Target content exceeds the maximum supported size (20 MB)
415 No Unsupported content type
429 No Per-minute rate limit exceeded; honor Retry-After
500 No Internal error

Parameters:

Returns:

See Also:



534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
# File 'lib/context_dev/resources/web.rb', line 534

def web_scrape_md(params)
  parsed, options = ContextDev::WebWebScrapeMdParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/markdown",
    query: query.transform_keys(
      exclude_selectors: "excludeSelectors",
      include_frames: "includeFrames",
      include_images: "includeImages",
      include_links: "includeLinks",
      include_selectors: "includeSelectors",
      max_age_ms: "maxAgeMs",
      settle_animations: "settleAnimations",
      shorten_base64_images: "shortenBase64Images",
      timeout_ms: "timeoutMS",
      use_main_content_only: "useMainContentOnly",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeMdResponse,
    options: options
  )
end

#web_scrape_sitemap(domain:, headers: nil, max_links: nil, search: nil, sitemap_url: nil, tags: nil, timeout_ms: nil, url_regex: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeSitemapResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeSitemapParams for more details.

Crawl an entire website's sitemap and return all discovered page URLs. Pass search to have the crawled sitemap filtered down to the pages about a phrase (for example pricing and plans or api authentication docs), most relevant first — a searched crawl scans the whole sitemap and costs 2 credits instead of 1.

Parameters:

  • domain (String)

    Domain to build a sitemap for

  • headers (Hash{Symbol=>String})

    Optional outbound HTTP headers forwarded only to the target URL, sent as deep-ob

  • max_links (Integer)

    Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Mi

  • search (String)

    Optional search phrase. When provided, the crawled sitemap is filtered to the pa

  • sitemap_url (String)

    Optional explicit sitemap URL. When provided, exactly this sitemap is crawled in

  • tags (Array<String>)

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer)

    Optional timeout in milliseconds for the request. If the request takes longer th

  • url_regex (String)

    Optional RE2-compatible regex pattern. Only URLs matching this pattern are retur

  • zdr (Symbol, ContextDev::Models::WebWebScrapeSitemapParams::Zdr)

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
# File 'lib/context_dev/resources/web.rb', line 592

def web_scrape_sitemap(params)
  parsed, options = ContextDev::WebWebScrapeSitemapParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/sitemap",
    query: query.transform_keys(
      max_links: "maxLinks",
      sitemap_url: "sitemapUrl",
      timeout_ms: "timeoutMS",
      url_regex: "urlRegex"
    ),
    model: ContextDev::Models::WebWebScrapeSitemapResponse,
    options: options
  )
end