Module: WebStruct::Http

Defined in:

lib/webstruct/http.rb,
lib/webstruct/http/url.rb,
lib/webstruct/http/mime.rb,
lib/webstruct/http/shell.rb,
lib/webstruct/page/content_type.rb

Overview

Internal Faraday-based HTTP request helper.

Defined Under Namespace

Modules: ContentType, Mime, Shell, Url

Constant Summary

FARADAY_OPTION_KEYS =

%i[max_redirects read_timeout open_timeout user_agent].freeze

DEFAULT_USER_AGENT =

Default User-Agent string used for HTTP requests.

"WebStruct/#{WebStruct::VERSION} (+https://rubygems.org/gems/webstruct)".freeze

DEFAULT_FARADAY_OPTIONS =

Default Faraday-related options merged into each get call.

Note: :max_redirects is passed to redirect middleware as :limit; timeouts are seconds.

{
  max_redirects: 5,
  read_timeout: 10,
  open_timeout: 5,
  user_agent: DEFAULT_USER_AGENT
}.freeze

Class Method Summary

• .get(url) ⇒ Page

  Requests a URL with Faraday, optionally runs Shell for HTML-like bodies, and returns a Page.

Class Method Details

.get(url) ⇒ Page

Note:

With the default Faraday adapter, the response body may be fully buffered before the size check runs; :max_body_bytes still prevents MIME resolution, shell detection, and Page construction for oversized bodies.

Requests a URL with Faraday, optionally runs Shell for HTML-like bodies, and returns a Page.

Parameters:

• url (String) —

  target URL (validated by WebStruct::Http::Url.verify!)

• options (Hash) —

  Faraday keys merged with DEFAULT_FARADAY_OPTIONS; :max_body_bytes when set (positive Integer), rejects bodies larger than that many bytes before parsing

Returns:

• (Page)

Raises:

• (InvalidUrlError) —

  from WebStruct::Http::Url.verify! when the URL is invalid

• (ArgumentError) —

  when :max_body_bytes is present but not a positive Integer

• (BodyTooLargeError) —

  when the response body exceeds :max_body_bytes

• (JavaScriptRequiredError) —

  from WebStruct::Http::Shell.detect! when the HTML looks like a JS shell

• (ParseError) —

  when JSON bodies are invalid

# File 'lib/webstruct/http.rb', line 43

def get(url, **)
  uri, max_body_bytes, faraday_opts, content_opts = get_options(url, **)
  response = faraday(faraday_opts).get(uri)
  body = read_body_within_byte_limit(response, max_body_bytes)
  type = Mime.resolve(response.headers["content-type"], body)

  Shell.detect!(body) if Mime.html_like?(type)

  Page.new(response, type, **content_opts)
end