Class: ContextDev::Resources::Web

Inherits:
Object
  • Object
show all
Defined in:
lib/context_dev/resources/web.rb

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ Web

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Web.

Parameters:



292
293
294
 
# File 'lib/context_dev/resources/web.rb', line 292

def initialize(client:)
  @client = client
end

Instance Method Details

#extract_fonts(direct_url: nil, domain: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractFontsResponse

Some parameter documentations has been truncated, see Models::WebExtractFontsParams for more details.

Scrape font information from a website including font families, usage statistics, fallbacks, and element/word counts.

Parameters:

  • direct_url (String) —

    A specific URL to fetch fonts from directly, bypassing domain resolution (e.g.,

  • domain (String) —

    Domain name to extract fonts from (e.g., ‘example.com’, ‘google.com’). The domai

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



25
26
27
28
29
30
31
32
33
34
35
 
# File 'lib/context_dev/resources/web.rb', line 25

def extract_fonts(params = {})
  parsed, options = ContextDev::WebExtractFontsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/fonts",
    query: query.transform_keys(direct_url: "directUrl", timeout_ms: "timeoutMS"),
    model: ContextDev::Models::WebExtractFontsResponse,
    options: options
  )
end

#extract_styleguide(direct_url: nil, domain: nil, timeout_ms: nil, request_options: {}) ⇒ ContextDev::Models::WebExtractStyleguideResponse

Some parameter documentations has been truncated, see Models::WebExtractStyleguideParams for more details.

Extract a comprehensive design system from a website including colors, typography, spacing, shadows, and UI components.

Parameters:

  • direct_url (String) —

    A specific URL to fetch the styleguide from directly, bypassing domain resolutio

  • domain (String) —

    Domain name to extract styleguide from (e.g., ‘example.com’, ‘google.com’). The

  • timeout_ms (Integer) —

    Optional timeout in milliseconds for the request. If the request takes longer th

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/context_dev/resources/web.rb', line 56

def extract_styleguide(params = {})
  parsed, options = ContextDev::WebExtractStyleguideParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/styleguide",
    query: query.transform_keys(direct_url: "directUrl", timeout_ms: "timeoutMS"),
    model: ContextDev::Models::WebExtractStyleguideResponse,
    options: options
  )
end

#screenshot(direct_url: nil, domain: nil, full_screenshot: nil, page: nil, prioritize: nil, request_options: {}) ⇒ ContextDev::Models::WebScreenshotResponse

Some parameter documentations has been truncated, see Models::WebScreenshotParams for more details.

Capture a screenshot of a website.

Parameters:

Returns:

See Also:



90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/context_dev/resources/web.rb', line 90

def screenshot(params = {})
  parsed, options = ContextDev::WebScreenshotParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/screenshot",
    query: query.transform_keys(direct_url: "directUrl", full_screenshot: "fullScreenshot"),
    model: ContextDev::Models::WebScreenshotResponse,
    options: options
  )
end

#web_crawl_md(url:, follow_subdomains: nil, include_frames: nil, include_images: nil, include_links: nil, max_age_ms: nil, max_depth: nil, max_pages: nil, parse_pdf: nil, shorten_base64_images: nil, url_regex: nil, use_main_content_only: nil, request_options: {}) ⇒ ContextDev::Models::WebWebCrawlMdResponse

Some parameter documentations has been truncated, see Models::WebWebCrawlMdParams for more details.

Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages.

Parameters:

  • url (String) —

    The starting URL for the crawl (must include http:// or https:// protocol)

  • follow_subdomains (Boolean) —

    When true, follow links on subdomains of the starting URL’s domain (e.g. docs.ex

  • include_frames (Boolean) —

    When true, the contents of iframes are rendered to Markdown for each crawled pag

  • include_images (Boolean) —

    Include image references in the Markdown output

  • include_links (Boolean) —

    Preserve hyperlinks in the Markdown output

  • max_age_ms (Integer) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • max_depth (Integer) —

    Maximum link depth from the starting URL (0 = only the starting page)

  • max_pages (Integer) —

    Maximum number of pages to crawl. Hard cap: 500.

  • parse_pdf (Boolean) —

    When true (default), PDF pages are fetched and their text layer is extracted and

  • shorten_base64_images (Boolean) —

    Truncate base64-encoded image data in the Markdown output

  • url_regex (String) —

    Regex pattern. Only URLs matching this pattern will be followed and scraped.

  • use_main_content_only (Boolean) —

    Extract only the main content, stripping headers, footers, sidebars, and navigat

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



139
140
141
142
143
144
145
146
147
148
 
# File 'lib/context_dev/resources/web.rb', line 139

def web_crawl_md(params)
  parsed, options = ContextDev::WebWebCrawlMdParams.dump_request(params)
  @client.request(
    method: :post,
    path: "web/crawl",
    body: parsed,
    model: ContextDev::Models::WebWebCrawlMdResponse,
    options: options
  )
end

#web_scrape_html(url:, include_frames: nil, max_age_ms: nil, parse_pdf: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeHTMLResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeHTMLParams for more details.

Scrapes the given URL and returns the raw HTML content of the page.

Parameters:

  • url (String) —

    Full URL to scrape (must include http:// or https:// protocol)

  • include_frames (Boolean) —

    When true, iframes are rendered inline into the returned HTML.

  • max_age_ms (Integer) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • parse_pdf (Boolean) —

    When true (default), PDF URLs are fetched and their text layer is extracted and

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
 
# File 'lib/context_dev/resources/web.rb', line 170

def web_scrape_html(params)
  parsed, options = ContextDev::WebWebScrapeHTMLParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/html",
    query: query.transform_keys(
      include_frames: "includeFrames",
      max_age_ms: "maxAgeMs",
      parse_pdf: "parsePDF"
    ),
    model: ContextDev::Models::WebWebScrapeHTMLResponse,
    options: options
  )
end

#web_scrape_images(url:, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeImagesResponse

Scrapes all images from the given URL. Extracts images from img, svg, picture/source, link, and video elements including inline SVGs, base64 data URIs, and standard URLs.

Parameters:

  • url (String) —

    Full URL to scrape images from (must include http:// or https:// protocol)

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



199
200
201
202
203
204
205
206
207
208
209
 
# File 'lib/context_dev/resources/web.rb', line 199

def web_scrape_images(params)
  parsed, options = ContextDev::WebWebScrapeImagesParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/images",
    query: query,
    model: ContextDev::Models::WebWebScrapeImagesResponse,
    options: options
  )
end

#web_scrape_md(url:, include_frames: nil, include_images: nil, include_links: nil, max_age_ms: nil, parse_pdf: nil, shorten_base64_images: nil, use_main_content_only: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeMdResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeMdParams for more details.

Scrapes the given URL into LLM usable Markdown.

Parameters:

  • url (String) —

    Full URL to scrape into LLM usable Markdown (must include http:// or https:// pr

  • include_frames (Boolean) —

    When true, the contents of iframes are rendered to Markdown.

  • include_images (Boolean) —

    Include image references in Markdown output

  • include_links (Boolean) —

    Preserve hyperlinks in Markdown output

  • max_age_ms (Integer) —

    Return a cached result if a prior scrape for the same parameters exists and is y

  • parse_pdf (Boolean) —

    When true (default), PDF URLs are fetched and their text layer is extracted and

  • shorten_base64_images (Boolean) —

    Shorten base64-encoded image data in the Markdown output

  • use_main_content_only (Boolean) —

    Extract only the main content of the page, excluding headers, footers, sidebars,

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
 
# File 'lib/context_dev/resources/web.rb', line 239

def web_scrape_md(params)
  parsed, options = ContextDev::WebWebScrapeMdParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/markdown",
    query: query.transform_keys(
      include_frames: "includeFrames",
      include_images: "includeImages",
      include_links: "includeLinks",
      max_age_ms: "maxAgeMs",
      parse_pdf: "parsePDF",
      shorten_base64_images: "shortenBase64Images",
      use_main_content_only: "useMainContentOnly"
    ),
    model: ContextDev::Models::WebWebScrapeMdResponse,
    options: options
  )
end

#web_scrape_sitemap(domain:, max_links: nil, url_regex: nil, request_options: {}) ⇒ ContextDev::Models::WebWebScrapeSitemapResponse

Some parameter documentations has been truncated, see Models::WebWebScrapeSitemapParams for more details.

Crawl an entire website’s sitemap and return all discovered page URLs.

Parameters:

  • domain (String) —

    Domain to build a sitemap for

  • max_links (Integer) —

    Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Mi

  • url_regex (String) —

    Optional RE2-compatible regex pattern. Only URLs matching this pattern are retur

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



277
278
279
280
281
282
283
284
285
286
287
 
# File 'lib/context_dev/resources/web.rb', line 277

def web_scrape_sitemap(params)
  parsed, options = ContextDev::WebWebScrapeSitemapParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "web/scrape/sitemap",
    query: query.transform_keys(max_links: "maxLinks", url_regex: "urlRegex"),
    model: ContextDev::Models::WebWebScrapeSitemapResponse,
    options: options
  )
end