Module: Pikuri::Tool::Fetch

Defined in:
lib/pikuri/tool/fetch.rb

Overview

Truncation policy and Tool spec for the fetch tool. HTTP work lives in Scraper.fetch; this wrapper accepts only textual content-types, applies a character cap, and exposes the result in OpenAI tool-call shape. Sister of WebScrape but with no extraction pass — bodies are returned verbatim, for raw textual data (JSON APIs, CSV, robots.txt, sitemaps, source files) where a rendering pass would corrupt the payload.

Constant Summary collapse

DEFAULT_MAX_CHARS =

Returns default character cap on the body from fetch. Smaller than WebScrape::DEFAULT_MAX_CHARS because fetch's content is bimodal — most JSON/XML/CSV is tiny, and large dumps are better re-requested deliberately than padded into every default.

Returns:

  • (Integer) —

    default character cap on the body from fetch. Smaller than WebScrape::DEFAULT_MAX_CHARS because fetch's content is bimodal — most JSON/XML/CSV is tiny, and large dumps are better re-requested deliberately than padded into every default.

5_000
MAX_MAX_CHARS =

Returns hard ceiling on fetch's max_chars; matches WebScrape::MAX_MAX_CHARS.

Returns:

100_000
TEXTUAL_APPLICATION_TYPES =

Application content-types that are textual in practice, so safe to return verbatim despite failing the text/* check. Anything outside text/* and this list is refused.

Returns:

  • (Array<String>) 
%w[
  application/json
  application/xml
  application/javascript
  application/xhtml+xml
  application/rss+xml
  application/atom+xml
].freeze
CACHE =

On-disk cache for fetch, in its own subdir so a fetch and a web_scrape on the same URL don't collide (raw body vs extracted Markdown). A method so specs can swap UrlCache::NULL.

Returns:

UrlCache.new(ttl: UrlCache::DEFAULT_TTL, dir: "#{UrlCache::ROOT_DIR}/fetch")

Class Method Summary collapse

Class Method Details

.cache ⇒ UrlCache, #fetch

Accessor for CACHE; specs swap in UrlCache::NULL or an isolated cache.

Returns:



44
45
46
 
# File 'lib/pikuri/tool/fetch.rb', line 44

def self.cache
  CACHE
end

.download(url) ⇒ String

GET url and verify the content-type is textual. Always hits the network (caller caches/truncates).

Parameters:

  • url (String)

Returns:

  • (String) —

    response body

Raises:

  • (Scraper::FetchError) —

    on HTTP non-2xx, network failure, redirect exhaustion, missing Location, or a non-textual content-type



77
78
79
80
81
82
83
84
 
# File 'lib/pikuri/tool/fetch.rb', line 77

def self.download(url)
  fetched = Scraper.fetch(url)
  return fetched.body if textual?(fetched.content_type)

  raise Scraper::FetchError,
        "refused to fetch #{url}: content-type #{fetched.content_type.inspect} " \
        'is not textual (use web_scrape for rendered pages)'
end

.fetch(url, max_chars: DEFAULT_MAX_CHARS) ⇒ String

Download url via Scraper.fetch and return the body verbatim, if the content-type is textual (any text/* plus TEXTUAL_APPLICATION_TYPES); anything else (PDF, image, binary) yields an "Error: ..." string. Cached on disk keyed by URL; max_chars is not in the key (values share one entry, truncation runs after the lookup). Cached only on success — Scraper::FetchError is caught outside cache.fetch, so failures are never persisted; other exceptions bubble up.

Parameters:

  • url (String) —

    absolute HTTP(S) URL

  • max_chars (Integer) (defaults to: DEFAULT_MAX_CHARS) —

    body cap, clamped to [1, {MAX_MAX_CHARS}]; default DEFAULT_MAX_CHARS. Over the cap, output is cut with a marker.

Returns:

  • (String) —

    body, possibly truncated, or "Error: ..." on a recoverable failure



62
63
64
65
66
67
68
 
# File 'lib/pikuri/tool/fetch.rb', line 62

def self.fetch(url, max_chars: DEFAULT_MAX_CHARS)
  max_chars = max_chars.clamp(1, MAX_MAX_CHARS)
  body = cache.fetch(url) { download(url) }
  truncate(body, max_chars)
rescue Scraper::FetchError => e
  "Error: #{e.message}"
end

.textual?(content_type) ⇒ Boolean

Returns true when text/* or in TEXTUAL_APPLICATION_TYPES.

Parameters:

  • content_type (String) —

    normalized content-type from Scraper.fetch

Returns:



88
89
90
91
 
# File 'lib/pikuri/tool/fetch.rb', line 88

def self.textual?(content_type)
  content_type.start_with?('text/') ||
    TEXTUAL_APPLICATION_TYPES.include?(content_type)
end

.truncate(body, max_chars) ⇒ String

Cut body to at most max_chars, appending a marker with the original length when truncation happens. Same shape as WebScrape.truncate.

Parameters:

  • body (String) —

    full response body

  • max_chars (Integer) —

    cap; assumed already clamped

Returns:

  • (String) 


99
100
101
102
103
104
105
 
# File 'lib/pikuri/tool/fetch.rb', line 99

def self.truncate(body, max_chars)
  return body if body.length <= max_chars

  "#{body[0, max_chars]}\n\n" \
    "... [truncated at #{max_chars} of #{body.length} chars; " \
    'call again with a larger `max_chars` to see more]'
end