Module: Pikuri::Tool::WebScrape

Defined in:
lib/pikuri/tool/web_scrape.rb

Overview

Truncation policy and Tool spec for the web_scrape tool. Scraping lives in Scraper; this wrapper applies a character cap and exposes the result in OpenAI tool-call shape.

Constant Summary collapse

DEFAULT_MAX_CHARS =

Returns default character cap on the Markdown from visit. Sized to cover most post-readability article bodies in full on the first call, so the LLM doesn't re-request and duplicate the prefix in its context. Genuinely long pages still get cut, and the marker invites a deliberate larger call.

Returns:

  • (Integer) —

    default character cap on the Markdown from visit. Sized to cover most post-readability article bodies in full on the first call, so the LLM doesn't re-request and duplicate the prefix in its context. Genuinely long pages still get cut, and the marker invites a deliberate larger call.

20_000
MAX_MAX_CHARS =

Returns hard ceiling on visit's max_chars; requests above it are clamped silently so the LLM can't dump an arbitrarily large page.

Returns:

  • (Integer) —

    hard ceiling on visit's max_chars; requests above it are clamped silently so the LLM can't dump an arbitrarily large page.

100_000
CACHE =

On-disk cache for visit. A method so specs can swap UrlCache::NULL.

Returns:

UrlCache.new(ttl: UrlCache::DEFAULT_TTL, dir: "#{UrlCache::ROOT_DIR}/web_scrape")

Class Method Summary collapse

Class Method Details

.cache ⇒ UrlCache, #fetch

Accessor for CACHE; specs swap in UrlCache::NULL or an isolated cache.

Returns:



27
28
29
 
# File 'lib/pikuri/tool/web_scrape.rb', line 27

def self.cache
  CACHE
end

.truncate(markdown, max_chars) ⇒ String

Cut markdown to at most max_chars, appending a marker with the original length when truncation happens.

Parameters:

  • markdown (String) —

    full Markdown text

  • max_chars (Integer) —

    cap; assumed already clamped

Returns:

  • (String) 


58
59
60
61
62
63
64
 
# File 'lib/pikuri/tool/web_scrape.rb', line 58

def self.truncate(markdown, max_chars)
  return markdown if markdown.length <= max_chars

  "#{markdown[0, max_chars]}\n\n" \
    "... [truncated at #{max_chars} of #{markdown.length} chars; " \
    'call again with a larger `max_chars` to see more]'
end

.visit(url, max_chars: DEFAULT_MAX_CHARS) ⇒ String

Fetch url via Scraper and truncate the rendered Markdown to max_chars. The full Markdown is cached on disk keyed by URL (repeat visits skip the network and extraction); max_chars is not in the key (values share one entry, truncation runs after the lookup). Scraper::FetchError is caught outside cache.fetch and returned as "Error: ..." (so the LLM can retry a different URL) — never persisted; other exceptions bubble up.

Parameters:

  • url (String) —

    absolute HTTP(S) URL

  • max_chars (Integer) (defaults to: DEFAULT_MAX_CHARS) —

    Markdown cap, clamped to [1, {MAX_MAX_CHARS}]; default DEFAULT_MAX_CHARS. Over the cap, output is cut with a marker.

Returns:

  • (String) —

    Markdown, possibly truncated, or "Error: ..." on a recoverable fetch failure



44
45
46
47
48
49
50
 
# File 'lib/pikuri/tool/web_scrape.rb', line 44

def self.visit(url, max_chars: DEFAULT_MAX_CHARS)
  max_chars = max_chars.clamp(1, MAX_MAX_CHARS)
  markdown = cache.fetch(url) { Scraper.visit(url) }
  truncate(markdown, max_chars)
rescue Scraper::FetchError => e
  "Error: #{e.message}"
end