Class: Pikuri::Extractors::Documents

Inherits:
Object
  • Object
show all
Defined in:
lib/pikuri/extractors/documents.rb

Overview

Document extractor for the Pikuri::Extractor registry: DOCX / ODT / XLSX / legacy XLS / PPTX / EPUB / RTF / PDF → Markdown, by piping the document bytes through pandoc (ODF, RTF, EPUB, DOCX), markitdown (OOXML spreadsheet / presentation), or pdftotext (PDF), selected per format.

Prefers a one-shot, networkless docker container (IMAGE) — bytes in via stdin, Markdown out via stdout, no volume mounts — and falls back to host pandoc / markitdown / pdftotext CLIs on the same stdin→stdout contract when docker is absent. Requiring the gem defines this class + the shared DOCUMENTS instance but registers nothing; a host opts in with DOCUMENTS.register.

The cross-cutting why — the container's security + reproducibility rationale, the PDF-arm trade-off against pikuri-pdf, the deliberately-unsupported formats (ODS/ODP, OCR/audio), and why paging re-converts each time — lives in pikuri-extractors/DESIGN.md.

Format detection

#matches? claims content by content-type (CONTENT_TYPES) or byte sniff (see #sniff). #extract re-sniffs (the registry duck type doesn't pass content_type to extract); when the sniff is blind — legacy XLS, an OLE2 container whose discriminating directory sits past the sample — bytes go to markitdown with no hint and its own detection takes over. Consequence: a local .xls (no transport content-type, sniff blind) isn't claimed at all and keeps today's binary refusal. Ordering edge: this instance sits after core's HTML, so a PDF served under a lying text/html header goes to HTML (pikuri-pdf front-inserts and wins that) — accepted, rare.

Constant Summary collapse

LOGGER =

Returns gem-wide diagnostics logger.

Returns:

  • (Logger) —

    gem-wide diagnostics logger.

Pikuri.logger_for('Extractors')
IMAGE =

Returns converter image tag. Version-tied so a gem upgrade rebuilds with the new pins; pikuri-internal- prefix matches the container-naming convention of the vectordb/memory supervisors.

Returns:

  • (String) —

    converter image tag. Version-tied so a gem upgrade rebuilds with the new pins; pikuri-internal- prefix matches the container-naming convention of the vectordb/memory supervisors.

"pikuri-internal-extractors:#{Pikuri::VERSION}"
DOCKER_DIR =

Returns absolute path to the shipped docker build context (Dockerfile + convert.sh).

Returns:

  • (String) —

    absolute path to the shipped docker build context (Dockerfile + convert.sh).

File.expand_path('../../../docker', __dir__)
CONVERT_TIMEOUT =

Returns coreutils-+timeout+ budget for one conversion. Generous — a huge PPTX through markitdown can take a while — but bounded, so a wedged converter can't hang the agent loop.

Returns:

  • (String) —

    coreutils-+timeout+ budget for one conversion. Generous — a huge PPTX through markitdown can take a while — but bounded, so a wedged converter can't hang the agent loop.

'300s'
AUTO =

Returns sentinel format meaning "let markitdown's magic-byte detection decide" — the fallback when content was claimed by content-type but the byte sniff is blind.

Returns:

  • (String) —

    sentinel format meaning "let markitdown's magic-byte detection decide" — the fallback when content was claimed by content-type but the byte sniff is blind.

'auto'
PDF =

Returns the PDF format tag. A constant because PDF is the one format whose output gets a post-processing pass (#pdf_page_lines restoring "--- Page N ---" markers), so page provenance survives whichever PDF extractor a host wires.

Returns:

  • (String) —

    the PDF format tag. A constant because PDF is the one format whose output gets a post-processing pass (#pdf_page_lines restoring "--- Page N ---" markers), so page provenance survives whichever PDF extractor a host wires.

'pdf'
CONTENT_TYPES =

Returns normalized content-type → format tag (the tag doubles as the container entrypoint's dispatch argument and pandoc's -f / markitdown's -x value).

Returns:

  • (Hash{String => String}) —

    normalized content-type → format tag (the tag doubles as the container entrypoint's dispatch argument and pandoc's -f / markitdown's -x value).

{
  'application/vnd.oasis.opendocument.text' => 'odt',
  'application/rtf' => 'rtf',
  'text/rtf' => 'rtf',
  'application/epub+zip' => 'epub',
  'application/pdf' => PDF,
  'application/vnd.openxmlformats-officedocument.wordprocessingml.document' => 'docx',
  'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' => 'xlsx',
  'application/vnd.ms-excel' => 'xls',
  'application/vnd.openxmlformats-officedocument.presentationml.presentation' => 'pptx'
}.freeze
HOST_CONVERTERS =

Returns format tag → host CLIs that can convert it, in preference order. Mirrors the container entrypoint's dispatch (+docker/convert.sh+) — keep the two in sync. pandoc leads where both could serve (DOCX, EPUB): its readers preserve more structure.

Returns:

  • (Hash{String => Array<Symbol>}) —

    format tag → host CLIs that can convert it, in preference order. Mirrors the container entrypoint's dispatch (+docker/convert.sh+) — keep the two in sync. pandoc leads where both could serve (DOCX, EPUB): its readers preserve more structure.

{
  'odt'  => %i[pandoc],
  'rtf'  => %i[pandoc],
  'epub' => %i[pandoc markitdown],
  'docx' => %i[pandoc markitdown],
  'xlsx' => %i[markitdown],
  'xls'  => %i[markitdown],
  'pptx' => %i[markitdown],
  PDF    => %i[pdftotext],
  AUTO   => %i[markitdown]
}.freeze
ZIP_MAGIC =

Returns zip local-file-header magic, shared by every OOXML / ODF / EPUB document.

Returns:

  • (String) —

    zip local-file-header magic, shared by every OOXML / ODF / EPUB document.

"PK\x03\x04".b
VERSION_PROBE_FLAGS =

Returns host-CLI name → the flag that makes it print a version and exit 0, where --version (the default probe, see #cli?) doesn't work: poppler's pdftotext parses --version as a filename and exits 1, but accepts -v.

Returns:

  • (Hash{String => String}) —

    host-CLI name → the flag that makes it print a version and exit 0, where --version (the default probe, see #cli?) doesn't work: poppler's pdftotext parses --version as a filename and exits 1, but accepts -v.

{ 'pdftotext' => '-v' }.freeze

Instance Method Summary collapse

Instance Method Details

#ensure_image! ⇒ void

This method returns an undefined value.

Build the converter image now if it isn't present — for host scripts that prefer paying the one-time build (pip install + apt, minutes) at boot rather than mid-conversation. Entirely optional: #extract builds lazily on first use otherwise.

Raises:

  • (Pikuri::Extractor::Error) —

    when docker is unavailable or the build fails.



188
189
190
191
192
193
 
# File 'lib/pikuri/extractors/documents.rb', line 188

def ensure_image!
  raise Pikuri::Extractor::Error, '`docker` is unavailable; cannot build the converter image' unless docker?

  image_ready!
  nil
end

#extract(io) ⇒ String

Convert the whole document behind io to one Markdown String. PDFs come back as one +"--- Page N ---"+-headed block per text-carrying page (see PDF); a fully scanned PDF extracts to the empty String — same contract as pikuri-pdf's extractor.

Parameters:

  • io (IO, StringIO) —

    seekable IO positioned at the start.

Returns:

  • (String) —

    Markdown-flavoured UTF-8 text.

Raises:

  • (Pikuri::Extractor::Error) —

    when no converter is available, the conversion exits non-zero, or it times out.



139
140
141
142
143
 
# File 'lib/pikuri/extractors/documents.rb', line 139

def extract(io)
  with_converted(io) do |file, format|
    format == PDF ? pdf_page_lines(file).to_a.join("\n") : file.read
  end
end

#extract_lines(io) ⇒ Enumerator<String>

Same content as #extract, as a stream of +chomp+ed lines off the converter's stdout Tempfile. The full conversion still runs up front (fired on first consumption), but neither the document nor the Markdown ever materialises as one String. The enumerator owns the Tempfile and deletes it when iteration ends.

Parameters:

  • io (IO, StringIO) —

    seekable IO positioned at the start; must remain open until the enumerator is consumed.

Returns:

  • (Enumerator<String>)

Raises:

  • (Pikuri::Extractor::Error) —

    as for #extract, raised on first consumption.



156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/pikuri/extractors/documents.rb', line 156

def extract_lines(io)
  Enumerator.new do |yielder|
    with_converted(io) do |file, format|
      if format == PDF
        pdf_page_lines(file).each { |line| yielder << line }
      else
        file.each_line { |line| yielder << line.chomp }
      end
    end
  end
end

#kind ⇒ Symbol

Returns kind tag carried on Extractor::Page#kind.

Returns:

  • (Symbol) —

    kind tag carried on Extractor::Page#kind.



114
115
116
 
# File 'lib/pikuri/extractors/documents.rb', line 114

def kind
  :document
end

#matches?(sample:, content_type:) ⇒ Boolean

Claim content this extractor can convert: a recognised content-type, or a positive byte sniff (see "Format detection" in the class docs).

Parameters:

  • sample (String) —

    leading bytes of the content.

  • content_type (String, nil) —

    normalized content-type, or nil when the transport carries none (local files).

Returns:

  • (Boolean) 


126
127
128
 
# File 'lib/pikuri/extractors/documents.rb', line 126

def matches?(sample:, content_type:)
  CONTENT_TYPES.key?(content_type) || !sniff(sample).nil?
end

#register ⇒ Documents

Plug this extractor into Pikuri::Extractor.registry, before the terminal Passthrough entry but after core's HTML and pikuri-pdf's front-inserted PDF (both keep winning their formats). Idempotent — a second call is a no-op.

Returns:



174
175
176
177
178
 
# File 'lib/pikuri/extractors/documents.rb', line 174

def register
  registry = Pikuri::Extractor.registry
  registry.insert(-2, self) unless registry.include?(self)
  self
end