Class: ContextDev::Resources::Web

Inherits:
Object
  • Object
show all
Defined in:
lib/context_dev/resources/web.rb,
sig/context_dev/resources/web.rbs

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ Web

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Web.

Parameters:



593
594
595
 
# File 'lib/context_dev/resources/web.rb', line 593

def initialize(client:)
  @client = client
end

Instance Method Details

#extract(schema:, url:, fact_check: nil, follow_subdomains: nil, include_frames: nil, instructions: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractResponse

Some parameter documentations has been truncated, see Models::WebExtractParams for more details.

Crawl a website, use the provided JSON Schema and instructions to prioritize relevant internal links, and extract structured data from the selected pages.

Parameters:

  • schema (Hash{Symbol=>Object}) —

    JSON Schema for the returned data object. TypeScript Zod users can pass a JSON S

  • url (String) —

    The starting website URL to crawl and extract from. Must include http:// or http

  • fact_check (Boolean) —

    When true, every returned value must be grounded in facts stated on the page; fi

  • follow_subdomains (Boolean) —

    When true, follow links on subdomains of the starting URL's domain.

  • include_frames (Boolean) —

    When true, iframe contents are included in Markdown before extraction.

  • instructions (String) —

    Optional extraction guidance, such as which facts to prioritize or how to interp

  • max_age_ms (Integer) —

    Return cached scrape results if a prior scrape for the same parameters is younge

  • max_depth (Integer) —

    Optional maximum link depth from the starting URL (0 = only the starting page).

  • max_pages (Integer) —

    Maximum number of pages to analyze for extraction. Hard cap: 50. Defaults to 5.

  • pdf (ContextDev::Models::WebExtractParams::Pdf)
  • settle_animations (Boolean) —

    When true, waits briefly for CSS and transition animations to settle before extr

  • stop_after_ms (Integer) —

    Soft time budget for the crawl in milliseconds. Min: 10000 (10s). Max: 110000 (1

  • tags (Array<String>) —

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • wait_for_ms (Integer) —

    Optional browser wait time in milliseconds after initial page load for each craw

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



49
50
51
52
53
54
55
56
57
58
 
# File 'lib/context_dev/resources/web.rb', line 49

def extract(params)
  parsed, options = ContextDev::WebExtractParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/extract",
    body: parsed,
    model: ContextDev::Models::WebExtractResponse,
    options: options
  )
end

#extract_competitors(domain:, num_competitors: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractCompetitorsResponse

Some parameter documentations has been truncated, see Models::WebExtractCompetitorsParams for more details.

Analyze a company's landing page and web search evidence to return direct competitors for the same product or market.

Parameters:

  • domain (String) —

    Company domain to analyze, such as stripe.com. Full http(s) URLs are accepted

  • num_competitors (Integer) —

    Exact number of direct competitors to return. Defaults to 5.

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/context_dev/resources/web.rb', line 81

def extract_competitors(params)
  parsed, options = ContextDev::WebExtractCompetitorsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/competitors",
    query: query.transform_keys(num_competitors: "numCompetitors", timeout_ms: "timeoutMS"),
    model: ContextDev::Models::WebExtractCompetitorsResponse,
    options: options
  )
end

#extract_fonts(direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractFontsResponse

Some parameter documentations has been truncated, see Models::WebExtractFontsParams for more details.

Scrape font information from a website including font families, usage statistics, fallbacks, and element/word counts.

Parameters:

  • direct_url (String) —

    A specific URL to fetch fonts from directly, bypassing domain resolution (e.g.,

  • domain (String) —

    Domain name to extract fonts from (e.g., 'example.com', 'google.com'). The domai

  • max_age_ms (Integer, nil) —

    Maximum age in milliseconds for cached brand data before the API performs a hard

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/context_dev/resources/web.rb', line 116

def extract_fonts(params = {})
  parsed, options = ContextDev::WebExtractFontsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/fonts",
    query: query.transform_keys(
      direct_url: "directUrl",
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS"
    ),
    model: ContextDev::Models::WebExtractFontsResponse,
    options: options
  )
end

#extract_styleguide(color_scheme: nil, direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractStyleguideResponse

Some parameter documentations has been truncated, see Models::WebExtractStyleguideParams for more details.

Extract a comprehensive design system from a website including colors, typography, spacing, shadows, and UI components.

Parameters:

  • color_scheme (Symbol, ContextDev::Models::WebExtractStyleguideParams::ColorScheme) —

    Optional browser color scheme to emulate for websites that respond to prefers-co

  • direct_url (String) —

    A specific URL to fetch the styleguide from directly, bypassing domain resolutio

  • domain (String) —

    Domain name to extract styleguide from (e.g., 'example.com', 'google.com'). The

  • max_age_ms (Integer, nil) —

    Maximum age in milliseconds for cached brand data before the API performs a hard

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/context_dev/resources/web.rb', line 157

def extract_styleguide(params = {})
  parsed, options = ContextDev::WebExtractStyleguideParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/styleguide",
    query: query.transform_keys(
      color_scheme: "colorScheme",
      direct_url: "directUrl",
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS"
    ),
    model: ContextDev::Models::WebExtractStyleguideResponse,
    options: options
  )
end

#screenshot(color_scheme: nil, country: nil, direct_url: nil, domain: nil, full_screenshot: nil, handle_cookie_popup: nil, max_age_ms: nil, page: nil, scroll_offset: nil, tags: nil, timeout_ms: nil, viewport: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebScreenshotResponse

Some parameter documentations has been truncated, see Models::WebScreenshotParams for more details.

Capture a screenshot of a website.

Parameters:

  • color_scheme (Symbol, ContextDev::Models::WebScreenshotParams::ColorScheme) —

    Optional parameter to choose the site's visual theme in the screenshot. Use 'lig

  • country (Symbol, ContextDev::Models::WebScreenshotParams::Country) —

    Two-letter ISO 3166-1 alpha-2 country code identifying a supported Context.dev r

  • direct_url (String) —

    A specific URL to screenshot directly, bypassing domain resolution (e.g., 'https

  • domain (String) —

    Domain name to take screenshot of (e.g., 'example.com', 'google.com'). The domai

  • full_screenshot (Symbol, ContextDev::Models::WebScreenshotParams::FullScreenshot) —

    Optional parameter to determine screenshot type. If 'true', takes a full page sc

  • handle_cookie_popup (Boolean, Symbol, ContextDev::Models::WebScreenshotParams::HandleCookiePopup) —

    Optional parameter to control cookie/consent popup handling. If 'true', we dismi

  • max_age_ms (Integer, nil) —

    Return a cached screenshot if a prior screenshot for the same parameters exists

  • page (Symbol, ContextDev::Models::WebScreenshotParams::Page) —

    Optional parameter to specify which page type to screenshot. If provided, the sy

  • scroll_offset (Integer, nil) —

    Optional vertical scroll offset in pixels for capturing a long page in viewport-

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • viewport (ContextDev::Models::WebScreenshotParams::Viewport) —

    Optional browser viewport dimensions for the screenshot. Defaults to 1920x1080.

  • wait_for_ms (Integer, nil) —

    Optional browser wait time in milliseconds after initial page load before taking

  • zdr (Symbol, ContextDev::Models::WebScreenshotParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
 
# File 'lib/context_dev/resources/web.rb', line 214

def screenshot(params = {})
  parsed, options = ContextDev::WebScreenshotParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/screenshot",
    query: query.transform_keys(
      color_scheme: "colorScheme",
      direct_url: "directUrl",
      full_screenshot: "fullScreenshot",
      handle_cookie_popup: "handleCookiePopup",
      max_age_ms: "maxAgeMs",
      scroll_offset: "scrollOffset",
      timeout_ms: "timeoutMS",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebScreenshotResponse,
    options: options
  )
end

#search(query:, country: nil, exclude_domains: nil, freshness: nil, include_domains: nil, markdown_options: nil, num_results: nil, query_fanout: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebSearchResponse

Some parameter documentations has been truncated, see Models::WebSearchParams for more details.

Search the web and optionally scrape each result to Markdown in one round-trip.

Parameters:

  • query (String) —

    Search query. Accepts natural language as well as Google-style search operators

  • country (Symbol, ContextDev::Models::WebSearchParams::Country) —

    Two-letter ISO 3166-1 alpha-2 country code to localize results to a specific cou

  • exclude_domains (Array<String>) —

    Blocklist — drop results from these domains. Example: ["pinterest.com", "reddit.

  • freshness (Symbol, ContextDev::Models::WebSearchParams::Freshness) —

    Restrict results to content published within this window.

  • include_domains (Array<String>) —

    Allowlist — only return results from these domains. Example: ["arxiv.org", "gith

  • markdown_options (ContextDev::Models::WebSearchParams::MarkdownOptions) —

    Inline Markdown scraping for each result. Set enabled: true to activate.

  • num_results (Integer) —

    Number of results to request and return (10–100). Defaults to 10.

  • query_fanout (Boolean) —

    Expand the query into multiple parallel variants for broader recall.

  • tags (Array<String>) —

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



267
268
269
270
271
272
273
274
275
276
 
# File 'lib/context_dev/resources/web.rb', line 267

def search(params)
  parsed, options = ContextDev::WebSearchParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/search",
    body: parsed,
    model: ContextDev::Models::WebSearchResponse,
    options: options
  )
end

#web_crawl_md(url:, country: nil, exclude_selectors: nil, follow_subdomains: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, url_regex: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebCrawlMdResponse

Some parameter documentations has been truncated, see Models::WebWebCrawlMdParams for more details.

Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages.

Parameters:

  • url (String) —

    The starting URL for the crawl (must include http:// or https:// protocol)

  • country (Symbol, ContextDev::Models::WebWebCrawlMdParams::Country) —

    Two-letter ISO 3166-1 alpha-2 country code identifying a supported Context.dev r

  • exclude_selectors (Array<String>) —

    CSS selectors to remove before each crawled page is converted to Markdown. Appli

  • follow_subdomains (Boolean) —

    When true, follow links on subdomains of the starting URL's domain (e.g. docs.ex

  • include_frames (Boolean) —

    When true, the contents of iframes are rendered to Markdown for each crawled pag

  • include_images (Boolean) —

    Include image references in the Markdown output

  • include_links (Boolean) —

    Preserve hyperlinks in the Markdown output

  • include_selectors (Array<String>) —

    CSS selectors. When provided, only matching HTML subtrees (and their descendants

  • max_age_ms (Integer) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • max_depth (Integer) —

    Maximum link depth from the starting URL (0 = only the starting page)

  • max_pages (Integer) —

    Maximum number of pages to crawl. Hard cap: 500.

  • pdf (ContextDev::Models::WebWebCrawlMdParams::Pdf) —

    PDF parsing controls. Use start/end to limit text extraction and embedded-image

  • settle_animations (Boolean) —

    When true, waits briefly for CSS and transition animations to settle before extr

  • shorten_base64_images (Boolean) —

    Truncate base64-encoded image data in the Markdown output

  • stop_after_ms (Integer) —

    Soft time budget for the crawl in milliseconds. After each scrape, the crawler c

  • tags (Array<String>) —

    Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • url_regex (String) —

    Regex pattern. Only URLs matching this pattern will be followed and scraped.

  • use_main_content_only (Boolean) —

    Extract only the main content, stripping headers, footers, sidebars, and navigat

  • wait_for_ms (Integer) —

    Optional browser wait time in milliseconds after initial page load for each craw

  • zdr (Symbol, ContextDev::Models::WebWebCrawlMdParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



333
334
335
336
337
338
339
340
341
342
 
# File 'lib/context_dev/resources/web.rb', line 333

def web_crawl_md(params)
  parsed, options = ContextDev::WebWebCrawlMdParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/crawl",
    body: parsed,
    model: ContextDev::Models::WebWebCrawlMdResponse,
    options: options
  )
end

#web_scrape_html(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeHTMLResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeHTMLParams for more details.

Scrapes the given URL and returns the raw HTML content of the page. The base request costs 1 credit; requests with browser actions cost 2 credits.

Parameters:

Returns:

See Also:



387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
 
# File 'lib/context_dev/resources/web.rb', line 387

def web_scrape_html(params)
  parsed, options = ContextDev::WebWebScrapeHTMLParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/html",
    query: query.transform_keys(
      exclude_selectors: "excludeSelectors",
      include_frames: "includeFrames",
      include_selectors: "includeSelectors",
      max_age_ms: "maxAgeMs",
      settle_animations: "settleAnimations",
      timeout_ms: "timeoutMS",
      use_main_content_only: "useMainContentOnly",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeHTMLResponse,
    options: options
  )
end

#web_scrape_images(url:, actions: nil, dedupe: nil, enrichment: nil, headers: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeImagesResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeImagesParams for more details.

Extract image assets from a web page, including standard URLs, inline SVGs, data URIs, responsive image sources, metadata, CSS backgrounds, video posters, and embeds. The base request costs 1 credit, or 2 credits with browser actions. When enrichment is enabled, the entire call costs 5 credits, including requests that also use actions.

Parameters:

Returns:

See Also:



442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
 
# File 'lib/context_dev/resources/web.rb', line 442

def web_scrape_images(params)
  parsed, options = ContextDev::WebWebScrapeImagesParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/images",
    query: query.transform_keys(
      max_age_ms: "maxAgeMs",
      timeout_ms: "timeoutMS",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeImagesResponse,
    options: options
  )
end

#web_scrape_md(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeMdResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeMdParams for more details.

Scrapes the given URL into LLM usable Markdown. Inspect key_metadata on JSON responses from a recognized API key; use error_code to distinguish stable failure categories.

Billing & errors

HTTP status Billed? Meaning
200 Yes — 1 credit, or 2 credits with actions Successful scrape, including a zero-length result when includeSelectors matched nothing
400 No Invalid input, skipped PDF, or the page could not be scraped
401 / 403 No Invalid/disabled key, insufficient permissions, or credits exhausted; inspect error_code
404 No Target page returned or fingerprinted as not found
408 No Request timed out
415 No Unsupported content type
429 No Per-minute rate limit exceeded; honor Retry-After
500 No Internal error

Parameters:

Returns:

See Also:



521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
 
# File 'lib/context_dev/resources/web.rb', line 521

def web_scrape_md(params)
  parsed, options = ContextDev::WebWebScrapeMdParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/markdown",
    query: query.transform_keys(
      exclude_selectors: "excludeSelectors",
      include_frames: "includeFrames",
      include_images: "includeImages",
      include_links: "includeLinks",
      include_selectors: "includeSelectors",
      max_age_ms: "maxAgeMs",
      settle_animations: "settleAnimations",
      shorten_base64_images: "shortenBase64Images",
      timeout_ms: "timeoutMS",
      use_main_content_only: "useMainContentOnly",
      wait_for_ms: "waitForMs"
    ),
    model: ContextDev::Models::WebWebScrapeMdResponse,
    options: options
  )
end

#web_scrape_sitemap(domain:, headers: nil, max_links: nil, sitemap_url: nil, tags: nil, timeout_ms: nil, url_regex: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeSitemapResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeSitemapParams for more details.

Crawl an entire website's sitemap and return all discovered page URLs.

Parameters:

  • domain (String) —

    Domain to build a sitemap for

  • headers (Hash{Symbol=>String}) —

    Optional outbound HTTP headers forwarded only to the target URL, sent as deep-ob

  • max_links (Integer) —

    Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Mi

  • sitemap_url (String) —

    Optional explicit sitemap URL. When provided, exactly this sitemap is crawled in

  • tags (Array<String>) —

    Optional comma-separated caller-defined tags for tracking this request. Tags are

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • url_regex (String) —

    Optional RE2-compatible regex pattern. Only URLs matching this pattern are retur

  • zdr (Symbol, ContextDev::Models::WebWebScrapeSitemapParams::Zdr) —

    Set to enabled to bypass shared caches and omit request and response content fro

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
 
# File 'lib/context_dev/resources/web.rb', line 573

def web_scrape_sitemap(params)
  parsed, options = ContextDev::WebWebScrapeSitemapParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/sitemap",
    query: query.transform_keys(
      max_links: "maxLinks",
      sitemap_url: "sitemapUrl",
      timeout_ms: "timeoutMS",
      url_regex: "urlRegex"
    ),
    model: ContextDev::Models::WebWebScrapeSitemapResponse,
    options: options
  )
end