Class: ContextDev::Models::WebWebCrawlMdParams

Inherits:
Internal::Type::BaseModel show all
Extended by:
Internal::Type::RequestParameters::Converter
Includes:
Internal::Type::RequestParameters
Defined in:
lib/context_dev/models/web_web_crawl_md_params.rb,
sig/context_dev/models/web_web_crawl_md_params.rbs

Overview

See Also:

  • ContextDev::Resources::Web#web_crawl_md

Defined Under Namespace

Modules: Country, Zdr Classes: Pdf

Constant Summary

Constants included from Internal::Type::RequestParameters

Internal::Type::RequestParameters::ContextDev

Instance Attribute Summary collapse

Attributes included from Internal::Type::RequestParameters

#request_options

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Internal::Type::RequestParameters::Converter

dump_request

Methods included from Internal::Type::RequestParameters

included

Methods inherited from Internal::Type::BaseModel

==, #==, #[], coerce, #deconstruct_keys, #deep_to_h, dump, fields, hash, #hash, inherited, inspect, #inspect, known_fields, optional, recursively_to_h, required, #to_h, #to_json, #to_s, to_sorbet_type, #to_yaml

Methods included from Internal::Type::Converter

#coerce, coerce, #dump, dump, #inspect, inspect, meta_info, new_coerce_state, type_info

Methods included from Internal::Util::SorbetRuntimeSupport

#const_missing, #define_sorbet_constant!, #sorbet_constant_defined?, #to_sorbet_type, to_sorbet_type

Constructor Details

#initialize(end_: nil, ocr: nil, should_parse: nil, start: nil) ⇒ Object

Some parameter documentations has been truncated, see Pdf for more details.

PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range.

Parameters:

  • end_ (Integer) (defaults to: nil) —

    Last 1-based PDF page to parse. When omitted, parsing ends at the last page. Mus

  • ocr (Boolean) (defaults to: nil) —

    When true, detect and OCR images embedded in the selected PDF pages, inserting r

  • should_parse (Boolean) (defaults to: nil) —

    When true, PDF pages are fetched and parsed. When false, PDF pages are skipped e

  • start (Integer) (defaults to: nil) —

    First 1-based PDF page to parse. When omitted, parsing starts at the first page.




# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 161

Instance Attribute Details

#country ⇒ Symbol, ...

Two-letter ISO 3166-1 alpha-2 country code identifying a supported Context.dev residential proxy exit location. Must be one of Context.dev's supported countries. When provided, Context.dev fetches the target page from that country.

Returns:



22
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 22

optional :country, enum: -> { ContextDev::WebWebCrawlMdParams::Country }

#exclude_selectors ⇒ Array<String>?

CSS selectors to remove before each crawled page is converted to Markdown. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: "nav", "footer", ".ad-banner", "[aria-hidden=true]".

Parameters:

  • (::Array[String])

Returns:

  • (Array<String>, nil) 


30
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 30

optional :exclude_selectors, ContextDev::Internal::Type::ArrayOf[String], api_name: :excludeSelectors

#follow_subdomains ⇒ Boolean?

When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent.

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


38
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 38

optional :follow_subdomains, ContextDev::Internal::Type::Boolean, api_name: :followSubdomains

#include_frames ⇒ Boolean?

When true, the contents of iframes are rendered to Markdown for each crawled page.

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


45
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 45

optional :include_frames, ContextDev::Internal::Type::Boolean, api_name: :includeFrames

#include_images ⇒ Boolean?

Include image references in the Markdown output

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


51
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 51

optional :include_images, ContextDev::Internal::Type::Boolean, api_name: :includeImages

#include_links ⇒ Boolean?

Preserve hyperlinks in the Markdown output

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


57
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 57

optional :include_links, ContextDev::Internal::Type::Boolean, api_name: :includeLinks

#include_selectors ⇒ Array<String>?

CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each crawled page is converted to Markdown. When omitted, the entire document is kept. Examples: "article.main", "#content", "[role=main]".

Parameters:

  • (::Array[String])

Returns:

  • (Array<String>, nil) 


66
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 66

optional :include_selectors, ContextDev::Internal::Type::ArrayOf[String], api_name: :includeSelectors

#max_age_ms ⇒ Integer?

Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh.

Parameters:

  • (Integer)

Returns:

  • (Integer, nil) 


74
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 74

optional :max_age_ms, Integer, api_name: :maxAgeMs

#max_depth ⇒ Integer?

Maximum link depth from the starting URL (0 = only the starting page)

Parameters:

  • (Integer)

Returns:

  • (Integer, nil) 


80
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 80

optional :max_depth, Integer, api_name: :maxDepth

#max_pages ⇒ Integer?

Maximum number of pages to crawl. Hard cap: 500.

Parameters:

  • (Integer)

Returns:

  • (Integer, nil) 


86
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 86

optional :max_pages, Integer, api_name: :maxPages

#pdf ⇒ ContextDev::Models::WebWebCrawlMdParams::Pdf?

PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range.

Returns:



93
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 93

optional :pdf, -> { ContextDev::WebWebCrawlMdParams::Pdf }

#settle_animations ⇒ Boolean?

When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages.

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


101
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 101

optional :settle_animations, ContextDev::Internal::Type::Boolean, api_name: :settleAnimations

#shorten_base64_images ⇒ Boolean?

Truncate base64-encoded image data in the Markdown output

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


107
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 107

optional :shorten_base64_images, ContextDev::Internal::Type::Boolean, api_name: :shortenBase64Images

#stop_after_ms ⇒ Integer?

Soft time budget for the crawl in milliseconds. After each scrape, the crawler checks the elapsed time and, if exceeded, returns the pages collected so far instead of continuing. Min: 10000 (10s). Max: 110000 (110s). Default: 80000 (80s).

Parameters:

  • (Integer)

Returns:

  • (Integer, nil) 


116
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 116

optional :stop_after_ms, Integer, api_name: :stopAfterMs

#tags ⇒ Array<String>?

Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.

Parameters:

  • (::Array[String])

Returns:

  • (Array<String>, nil) 


122
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 122

optional :tags, ContextDev::Internal::Type::ArrayOf[String]

#timeout_ms ⇒ Integer?

Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).

Parameters:

  • (Integer)

Returns:

  • (Integer, nil) 


130
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 130

optional :timeout_ms, Integer, api_name: :timeoutMS

#url ⇒ String

The starting URL for the crawl (must include http:// or https:// protocol)

Parameters:

  • value (String)

Returns:

  • (String) 


14
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 14

required :url, String

#url_regex ⇒ String?

Regex pattern. Only URLs matching this pattern will be followed and scraped.

Parameters:

  • (String)

Returns:

  • (String, nil) 


136
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 136

optional :url_regex, String, api_name: :urlRegex

#use_main_content_only ⇒ Boolean?

Extract only the main content, stripping headers, footers, sidebars, and navigation

Parameters:

  • (Boolean)

Returns:

  • (Boolean, nil) 


143
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 143

optional :use_main_content_only, ContextDev::Internal::Type::Boolean, api_name: :useMainContentOnly

#wait_for_ms ⇒ Integer?

Optional browser wait time in milliseconds after initial page load for each crawled page. Min: 0. Max: 30000 (30 seconds).

Parameters:

  • (Integer)

Returns:

  • (Integer, nil) 


150
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 150

optional :wait_for_ms, Integer, api_name: :waitForMs

#zdr ⇒ Symbol, ...

Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.

Returns:



159
 
# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 159

optional :zdr, enum: -> { ContextDev::WebWebCrawlMdParams::Zdr }

Class Method Details

.values ⇒ Array<Symbol>

Returns:

  • (Array<Symbol>) 



# File 'lib/context_dev/models/web_web_crawl_md_params.rb', line 420

Instance Method Details

#to_hash ⇒ {

Returns:

  • ({) 


146
 
# File 'sig/context_dev/models/web_web_crawl_md_params.rbs', line 146

def to_hash: -> {