Class: ContextDev::Resources::Web
- Inherits:
-
Object
- Object
- ContextDev::Resources::Web
- Defined in:
- lib/context_dev/resources/web.rb,
sig/context_dev/resources/web.rbs
Instance Method Summary collapse
-
#extract(schema:, url:, fact_check: nil, follow_subdomains: nil, include_frames: nil, instructions: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractResponse
Some parameter documentations has been truncated, see Models::WebExtractParams for more details.
-
#extract_competitors(domain:, num_competitors: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractCompetitorsResponse
Some parameter documentations has been truncated, see Models::WebExtractCompetitorsParams for more details.
-
#extract_fonts(direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractFontsResponse
Some parameter documentations has been truncated, see Models::WebExtractFontsParams for more details.
-
#extract_styleguide(color_scheme: nil, direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractStyleguideResponse
Some parameter documentations has been truncated, see Models::WebExtractStyleguideParams for more details.
-
#initialize(client:) ⇒ Web
constructor
private
A new instance of Web.
-
#screenshot(color_scheme: nil, country: nil, direct_url: nil, domain: nil, full_screenshot: nil, handle_cookie_popup: nil, max_age_ms: nil, page: nil, scroll_offset: nil, tags: nil, timeout_ms: nil, viewport: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebScreenshotResponse
Some parameter documentations has been truncated, see Models::WebScreenshotParams for more details.
-
#search(query:, country: nil, exclude_domains: nil, freshness: nil, include_domains: nil, markdown_options: nil, num_results: nil, query_fanout: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebSearchResponse
Some parameter documentations has been truncated, see Models::WebSearchParams for more details.
-
#web_crawl_md(url:, country: nil, exclude_selectors: nil, follow_subdomains: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, url_regex: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebCrawlMdResponse
Some parameter documentations has been truncated, see Models::WebWebCrawlMdParams for more details.
-
#web_scrape_html(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeHTMLResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeHTMLParams for more details.
-
#web_scrape_images(url:, actions: nil, dedupe: nil, enrichment: nil, headers: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeImagesResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeImagesParams for more details.
-
#web_scrape_md(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeMdResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeMdParams for more details.
-
#web_scrape_sitemap(domain:, headers: nil, max_links: nil, search: nil, sitemap_url: nil, tags: nil, timeout_ms: nil, url_regex: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeSitemapResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeSitemapParams for more details.
Constructor Details
#initialize(client:) ⇒ Web
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Returns a new instance of Web.
612 613 614 |
# File 'lib/context_dev/resources/web.rb', line 612 def initialize(client:) @client = client end |
Instance Method Details
#extract(schema:, url:, fact_check: nil, follow_subdomains: nil, include_frames: nil, instructions: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractResponse
Some parameter documentations has been truncated, see Models::WebExtractParams for more details.
Crawl a website, use the provided JSON Schema and instructions to prioritize relevant internal links, and extract structured data from the selected pages.
49 50 51 52 53 54 55 56 57 58 |
# File 'lib/context_dev/resources/web.rb', line 49 def extract(params) parsed, = ContextDev::WebExtractParams.dump_request(params) @client.request( method: :post, path: "web/extract", body: parsed, model: ContextDev::Models::WebExtractResponse, options: ) end |
#extract_competitors(domain:, num_competitors: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractCompetitorsResponse
Some parameter documentations has been truncated, see Models::WebExtractCompetitorsParams for more details.
Analyze a company's landing page and web search evidence to return direct competitors for the same product or market.
81 82 83 84 85 86 87 88 89 90 91 |
# File 'lib/context_dev/resources/web.rb', line 81 def extract_competitors(params) parsed, = ContextDev::WebExtractCompetitorsParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/competitors", query: query.transform_keys(num_competitors: "numCompetitors", timeout_ms: "timeoutMS"), model: ContextDev::Models::WebExtractCompetitorsResponse, options: ) end |
#extract_fonts(direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractFontsResponse
Some parameter documentations has been truncated, see Models::WebExtractFontsParams for more details.
Scrape font information from a website including font families, usage statistics, fallbacks, and element/word counts.
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
# File 'lib/context_dev/resources/web.rb', line 116 def extract_fonts(params = {}) parsed, = ContextDev::WebExtractFontsParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/fonts", query: query.transform_keys( direct_url: "directUrl", max_age_ms: "maxAgeMs", timeout_ms: "timeoutMS" ), model: ContextDev::Models::WebExtractFontsResponse, options: ) end |
#extract_styleguide(color_scheme: nil, direct_url: nil, domain: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractStyleguideResponse
Some parameter documentations has been truncated, see Models::WebExtractStyleguideParams for more details.
Extract a comprehensive design system from a website including colors, typography, spacing, shadows, and UI components.
157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 |
# File 'lib/context_dev/resources/web.rb', line 157 def extract_styleguide(params = {}) parsed, = ContextDev::WebExtractStyleguideParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/styleguide", query: query.transform_keys( color_scheme: "colorScheme", direct_url: "directUrl", max_age_ms: "maxAgeMs", timeout_ms: "timeoutMS" ), model: ContextDev::Models::WebExtractStyleguideResponse, options: ) end |
#screenshot(color_scheme: nil, country: nil, direct_url: nil, domain: nil, full_screenshot: nil, handle_cookie_popup: nil, max_age_ms: nil, page: nil, scroll_offset: nil, tags: nil, timeout_ms: nil, viewport: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebScreenshotResponse
Some parameter documentations has been truncated, see Models::WebScreenshotParams for more details.
Capture a screenshot of a website.
214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 |
# File 'lib/context_dev/resources/web.rb', line 214 def screenshot(params = {}) parsed, = ContextDev::WebScreenshotParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/screenshot", query: query.transform_keys( color_scheme: "colorScheme", direct_url: "directUrl", full_screenshot: "fullScreenshot", handle_cookie_popup: "handleCookiePopup", max_age_ms: "maxAgeMs", scroll_offset: "scrollOffset", timeout_ms: "timeoutMS", wait_for_ms: "waitForMs" ), model: ContextDev::Models::WebScreenshotResponse, options: ) end |
#search(query:, country: nil, exclude_domains: nil, freshness: nil, include_domains: nil, markdown_options: nil, num_results: nil, query_fanout: nil, tags: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebSearchResponse
Some parameter documentations has been truncated, see Models::WebSearchParams for more details.
Search the web and optionally scrape each result to Markdown in one round-trip.
267 268 269 270 271 272 273 274 275 276 |
# File 'lib/context_dev/resources/web.rb', line 267 def search(params) parsed, = ContextDev::WebSearchParams.dump_request(params) @client.request( method: :post, path: "web/search", body: parsed, model: ContextDev::Models::WebSearchResponse, options: ) end |
#web_crawl_md(url:, country: nil, exclude_selectors: nil, follow_subdomains: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, stop_after_ms: nil, tags: nil, timeout_ms: nil, url_regex: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebCrawlMdResponse
Some parameter documentations has been truncated, see Models::WebWebCrawlMdParams for more details.
Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages.
333 334 335 336 337 338 339 340 341 342 |
# File 'lib/context_dev/resources/web.rb', line 333 def web_crawl_md(params) parsed, = ContextDev::WebWebCrawlMdParams.dump_request(params) @client.request( method: :post, path: "web/crawl", body: parsed, model: ContextDev::Models::WebWebCrawlMdResponse, options: ) end |
#web_scrape_html(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeHTMLResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeHTMLParams for more details.
Scrapes the given URL and returns the raw HTML content of the page. The base request costs 1 credit; requests with browser actions cost 2 credits.
387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 |
# File 'lib/context_dev/resources/web.rb', line 387 def web_scrape_html(params) parsed, = ContextDev::WebWebScrapeHTMLParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/scrape/html", query: query.transform_keys( exclude_selectors: "excludeSelectors", include_frames: "includeFrames", include_selectors: "includeSelectors", max_age_ms: "maxAgeMs", settle_animations: "settleAnimations", timeout_ms: "timeoutMS", use_main_content_only: "useMainContentOnly", wait_for_ms: "waitForMs" ), model: ContextDev::Models::WebWebScrapeHTMLResponse, options: ) end |
#web_scrape_images(url:, actions: nil, dedupe: nil, enrichment: nil, headers: nil, max_age_ms: nil, tags: nil, timeout_ms: nil, wait_for_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeImagesResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeImagesParams for more details.
Extract image assets from a web page, including standard URLs, inline SVGs, data URIs, responsive image sources, metadata, CSS backgrounds, video posters, and embeds. The base request costs 1 credit, or 2 credits with browser actions. When enrichment is enabled, the entire call costs 5 credits, including requests that also use actions.
442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 |
# File 'lib/context_dev/resources/web.rb', line 442 def web_scrape_images(params) parsed, = ContextDev::WebWebScrapeImagesParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/scrape/images", query: query.transform_keys( max_age_ms: "maxAgeMs", timeout_ms: "timeoutMS", wait_for_ms: "waitForMs" ), model: ContextDev::Models::WebWebScrapeImagesResponse, options: ) end |
#web_scrape_md(url:, actions: nil, country: nil, exclude_selectors: nil, headers: nil, include_frames: nil, include_images: nil, include_links: nil, include_selectors: nil, max_age_ms: nil, pdf: nil, settle_animations: nil, shorten_base64_images: nil, tags: nil, timeout_ms: nil, use_main_content_only: nil, wait_for_ms: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeMdResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeMdParams for more details.
Scrapes the given URL into LLM usable Markdown. Inspect key_metadata on JSON responses from a recognized API key; use error_code to distinguish stable failure categories.
YouTube
YouTube URLs return the video or channel itself rather than the surrounding
player and navigation chrome. A URL addressing a single video (/watch,
youtu.be, /shorts, /embed, /live) returns its title, channel, duration,
view count, keywords, full description, and the transcript when the video has
captions that can be retrieved; videos without captions return everything except
the transcript. A channel URL (/channel/UC…, /@handle, /c/…, /user/…)
returns its name, handle, subscriber count, video count, and full description.
When includeImages=true, video responses also include the thumbnail and
channel responses include the avatar. Costs the same as any other scrape.
Billing & errors
| HTTP status | Billed? | Meaning |
|---|---|---|
| 200 | Yes — 1 credit, or 2 credits with actions | Successful scrape, including a zero-length result when includeSelectors matched nothing |
| 400 | No | Invalid input, skipped PDF, or the page could not be scraped |
| 401 / 403 | No | Invalid/disabled key, insufficient permissions, or credits exhausted; inspect error_code |
| 404 | No | Target page returned or fingerprinted as not found |
| 408 | No | Request timed out |
| 413 | No | Target content exceeds the maximum supported size (20 MB) |
| 415 | No | Unsupported content type |
| 429 | No | Per-minute rate limit exceeded; honor Retry-After |
| 500 | No | Internal error |
534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 |
# File 'lib/context_dev/resources/web.rb', line 534 def web_scrape_md(params) parsed, = ContextDev::WebWebScrapeMdParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/scrape/markdown", query: query.transform_keys( exclude_selectors: "excludeSelectors", include_frames: "includeFrames", include_images: "includeImages", include_links: "includeLinks", include_selectors: "includeSelectors", max_age_ms: "maxAgeMs", settle_animations: "settleAnimations", shorten_base64_images: "shortenBase64Images", timeout_ms: "timeoutMS", use_main_content_only: "useMainContentOnly", wait_for_ms: "waitForMs" ), model: ContextDev::Models::WebWebScrapeMdResponse, options: ) end |
#web_scrape_sitemap(domain:, headers: nil, max_links: nil, search: nil, sitemap_url: nil, tags: nil, timeout_ms: nil, url_regex: nil, zdr: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeSitemapResponse
Some parameter documentations has been truncated, see Models::WebWebScrapeSitemapParams for more details.
Crawl an entire website's sitemap and return all discovered page URLs. Pass
search to have the crawled sitemap filtered down to the pages about a phrase
(for example pricing and plans or api authentication docs), most relevant
first — a searched crawl scans the whole sitemap and costs 2 credits instead
of 1.
592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 |
# File 'lib/context_dev/resources/web.rb', line 592 def web_scrape_sitemap(params) parsed, = ContextDev::WebWebScrapeSitemapParams.dump_request(params) query = ContextDev::Internal::Util.encode_query_params(parsed) @client.request( method: :get, path: "web/scrape/sitemap", query: query.transform_keys( max_links: "maxLinks", sitemap_url: "sitemapUrl", timeout_ms: "timeoutMS", url_regex: "urlRegex" ), model: ContextDev::Models::WebWebScrapeSitemapResponse, options: ) end |