Module: Pikuri::Extractor

Defined in:
lib/pikuri/extractor.rb,
lib/pikuri/extractor/html.rb,
lib/pikuri/extractor/passthrough.rb

Overview

The format→text extraction seam: one ordered registry of extractors turning an IO of a recognised format (HTML + plain text in core; PDF / office formats via the pikuri-pdf / pikuri-extractors gems) into Markdown-flavoured UTF-8, behind two front doors:

Tool::Scraper (dispatching on the HTTP Content-Type) and FileType (resolving local paths) both route through this registry, so "support a new format" is a registry entry, not a new dispatcher arm.

The extractor duck type

Each Extractor.registry entry implements:

  • matches?(sample:, content_type:) → Boolean — claim the content. sample is the leading FileType::SAMPLE_BYTES (magic-byte sniff); content_type is the normalized HTTP/FileType.detect_mime type, or nil ("no metadata — sniff if you can").
  • extract(io) → String — whole document as Markdown-flavoured UTF-8. Raises Error on content it claimed but can't parse.
  • kind → Symbol — a short tag (+:text+ / :pdf / :html) carried on Page#kind so callers word format-specific trailers without re-sniffing.

plus one optional method:

  • extract_lines(io) → Enumerator<String> — the same content as a lazy stream of +chomp+ed lines. Extractor.extract_paged prefers it and stops the moment the window fills, so the document's tail is never parsed (pikuri-pdf parses pages on access). Must be consumed while io is open; may raise Error mid-iteration. Extractors needing the whole document (HTML, any subprocess-based one) omit it, and Extractor.extract_paged extracts in full then windows.

Windowing (offset / limit / byte cap / truncation) is presentation and lives once in Extractor.extract_paged, never per extractor — extract_lines is line production, the only format-specific half of paging.

Errors

Both modes are failures the caller's LLM can react to, under one rescuable root:

Defined Under Namespace

Modules: HTML, Passthrough Classes: Page

Constant Summary collapse

Error =

Raised when an extractor claims content but fails to parse it (e.g. a malformed PDF). Message is LLM-presentable.

Class.new(StandardError)
Unsupported =

Raised by extract / extract_paged when no registry entry claims the content. Subclass of Error so callers that don't care about the distinction rescue one class.

Class.new(Error)
PAGE_DEFAULT_LIMIT =

Returns default line-window size for extract_paged when the caller omits limit.

Returns:

  • (Integer) —

    default line-window size for extract_paged when the caller omits limit.

2000
PAGE_MAX_BYTES =

Returns default hard byte cap on the content collected by a single extract_paged call. Bypassable by paging via offset. The rendered output is slightly larger (line numbering, trailer) — that's the caller's concern.

Returns:

  • (Integer) —

    default hard byte cap on the content collected by a single extract_paged call. Bypassable by paging via offset. The rendered output is slightly larger (line numbering, trailer) — that's the caller's concern.

50 * 1024
PAGE_MAX_LINE_LENGTH =

Returns default per-line character cap; extract_paged truncates longer lines and appends PAGE_LINE_TRUNCATION_MARKER.

Returns:

2000
PAGE_LINE_TRUNCATION_MARKER =

Returns suffix appended to a line truncated at PAGE_MAX_LINE_LENGTH.

Returns:

"... (line truncated to #{PAGE_MAX_LINE_LENGTH} chars)"

Class Method Summary collapse

Class Method Details

.extract(io, content_type: nil) ⇒ String

Extract the whole document behind io as one Markdown-flavoured UTF-8 String. May be empty (empty text file, scanned-image PDF).

Parameters:

  • io (IO, StringIO) —

    seekable IO at the content start; reads a leading sample for the sniff and rewinds before extracting.

  • content_type (String, nil) (defaults to: nil) —

    normalized content-type when the transport supplies one; nil when unknown (extractors sniff).

Returns:

  • (String)

Raises:

  • (Unsupported) —

    when no registry entry claims the content.

  • (Error) —

    when the matched extractor cannot parse it.



129
130
131
 
# File 'lib/pikuri/extractor.rb', line 129

def extract(io, content_type: nil)
  extractor_for(io, content_type).extract(io)
end

.extract_paged(io, content_type: nil, offset: 1, limit: PAGE_DEFAULT_LIMIT, max_bytes: PAGE_MAX_BYTES, max_line_length: PAGE_MAX_LINE_LENGTH) ⇒ Page

Extract io into a windowed Page: lines from offset (1-indexed) up to limit, stopping early at max_bytes, over-long lines truncated at max_line_length.

Lazy where the format allows: extract_lines extractors (text, pikuri-pdf) are consumed only until the window fills — a 500-page PDF's first window parses a handful of pages. Extractors without it (HTML) are extracted in full then windowed, which is why their total_lines is always exact.

Parameters:

  • io (IO, StringIO) —

    seekable IO at the start.

  • content_type (String, nil) (defaults to: nil) —

    as for extract.

  • offset (Integer) (defaults to: 1) —

    1-indexed first line; caller validates >= 1.

  • limit (Integer) (defaults to: PAGE_DEFAULT_LIMIT) —

    max lines to collect; caller validates >= 1.

  • max_bytes (Integer) (defaults to: PAGE_MAX_BYTES) —

    hard byte cap on collected content.

  • max_line_length (Integer) (defaults to: PAGE_MAX_LINE_LENGTH) —

    per-line truncation threshold.

Returns:

Raises:

  • (Unsupported) —

    when no registry entry claims the content.

  • (Error) —

    when the matched extractor cannot parse it.



152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/pikuri/extractor.rb', line 152

def extract_paged(io, content_type: nil, offset: 1, limit: PAGE_DEFAULT_LIMIT,
                  max_bytes: PAGE_MAX_BYTES, max_line_length: PAGE_MAX_LINE_LENGTH)
  extractor = extractor_for(io, content_type)
  if extractor.respond_to?(:extract_lines)
    # count_tail is a per-format economics call: once the window fills,
    # counting the rest of a text stream is a cheap sequential read (so
    # the trailer says "of N"), but for a PDF it would parse every
    # remaining page — what extract_lines exists to avoid. Plugged-in
    # extractors get the conservative default (stop early, total unknown).
    window(extractor.extract_lines(io),
           offset: offset, limit: limit, max_bytes: max_bytes,
           max_line_length: max_line_length, kind: extractor.kind,
           known_total: nil, count_tail: extractor.equal?(Passthrough))
  else
    lines = extractor.extract(io).split("\n")
    window(lines, offset: offset, limit: limit, max_bytes: max_bytes,
                  max_line_length: max_line_length, kind: extractor.kind,
                  known_total: lines.length)
  end
end

.registry ⇒ Array<#matches?>

The extractor registry, consulted in order — first match wins. Core ships HTML (content-type) then the terminal Passthrough plain-text arm. A gem picks its insertion point by claim strength: a magic-byte sniff that never misfires on text goes at the front to beat HTML under a lying header (+registry.unshift(X)+ — pikuri-pdf); a content-type/weaker claimer inserts before the terminal entry (+registry.insert(-2, X)+ — pikuri-extractors).

Returns:

  • (Array<#matches?>) —

    mutable, deliberately — the plug-in seam.



115
116
117
 
# File 'lib/pikuri/extractor.rb', line 115

def registry
  @registry ||= [HTML, Passthrough]
end