Module: Pikuri::FileType

Defined in:
lib/pikuri/file_type.rb

Overview

Magic-byte content sniffing, plus the path-aware front over the Extractor registry. Two core responsibilities:

  • FileType.detect_mime — recognise a file from its leading bytes. Returns a MIME String for formats pikuri handles specially (+application/pdf+, the four image formats), or nil for "unrecognised".
  • FileType.binary? — text-vs-binary heuristic, independent of FileType.detect_mime (a file can be both recognised, e.g. PDF, and binary): what the bytes are vs. whether they're safe to render as text.

On top sit two Pathname conveniences, FileType.read_as_text (whole document) and FileType.read_as_text_paged (line-windowed). Both own the path-level refusals (missing file, directory, image) and the exception mapping, then hand the IO to Extractor; format-to-text is the registry's business, so a new extractor extends these wrappers for free.

FileType.detect_mime/FileType.binary? take either a String of bytes (caller sampled) or a Pathname (opened binary, SAMPLE_BYTES read).

Deliberate non-goals

  • Not a full MIME database. The set grows when a tool needs a format, not speculatively.
  • No path/extension fallback. Extensions lie (a renamed .png); magic-byte detection on content is the source of truth.
  • No +image?+/+pdf?+ predicates. Callers do mime == 'application/pdf' — one character, zero added surface.

Constant Summary collapse

SAMPLE_BYTES =

Returns bytes to sample for detect_mime/binary?. Covers every prefix sniffed today (largest: WebP's 12-byte header) with slack.

Returns:

  • (Integer) —

    bytes to sample for detect_mime/binary?. Covers every prefix sniffed today (largest: WebP's 12-byte header) with slack.

4096
BINARY_NONPRINTABLE_THRESHOLD =

Returns fraction of the sample that may be non-printable before binary? flags the bytes as binary. Matches opencode's threshold.

Returns:

  • (Float) —

    fraction of the sample that may be non-printable before binary? flags the bytes as binary. Matches opencode's threshold.

0.30
IMAGE_MAGIC_BYTES =

Returns magic-byte prefixes → MIME types for the image formats with flat (offset-zero, fixed-length) signatures. WebP isn't here — its signature is split across the RIFF container header — and is handled directly in detect_mime.

Returns:

  • (Hash{String => String}) —

    magic-byte prefixes → MIME types for the image formats with flat (offset-zero, fixed-length) signatures. WebP isn't here — its signature is split across the RIFF container header — and is handled directly in detect_mime.

{
  "\x89PNG\r\n\x1a\n".b => 'image/png',
  "\xff\xd8\xff".b      => 'image/jpeg',
  "GIF87a".b            => 'image/gif',
  "GIF89a".b            => 'image/gif'
}.freeze
PDF_MAGIC =

Returns PDF magic prefix. Every conformant PDF starts with this five-byte ASCII sequence per ISO 32000-1 §7.5.2.

Returns:

  • (String) —

    PDF magic prefix. Every conformant PDF starts with this five-byte ASCII sequence per ISO 32000-1 §7.5.2.

'%PDF-'

Class Method Summary collapse

Class Method Details

.binary?(input) ⇒ Boolean

Text-vs-binary heuristic (opencode's): any NUL forces true; else ratio bytes outside printable (+\t\n\v\f\r+ + ASCII 32..126) against the sample. UTF-8 continuation bytes (0x80-0xBF) sit above 127 so they pass unflagged; an empty sample is not-binary.

Parameters:

  • input (String, Pathname) —

    bytes, or a Pathname opened binary and read up to SAMPLE_BYTES. Caller verifies the path exists.

Returns:

  • (Boolean) 


89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/pikuri/file_type.rb', line 89

def binary?(input)
  bytes = sample_of(input)
  return false if bytes.empty?

  non_printable = 0
  bytes.each_byte do |b|
    return true if b.zero?

    non_printable += 1 if b < 9 || (b > 13 && b < 32)
  end
  non_printable.to_f / bytes.bytesize > BINARY_NONPRINTABLE_THRESHOLD
end

.detect_mime(input) ⇒ String?

Recognise a file from its leading bytes. Returns the MIME String for formats pikuri handles specially, or nil for "unrecognised" (text, opaque binary, ...).

Parameters:

  • input (String, Pathname) —

    bytes to inspect, or a Pathname opened binary and read up to SAMPLE_BYTES. Caller verifies the path exists; missing-file errors propagate as Errno::ENOENT.

Returns:

  • (String, nil) 


67
68
69
70
71
72
73
74
75
76
77
78
79
 
# File 'lib/pikuri/file_type.rb', line 67

def detect_mime(input)
  bytes = sample_of(input)
  return 'application/pdf' if bytes.start_with?(PDF_MAGIC)

  IMAGE_MAGIC_BYTES.each do |prefix, mime|
    return mime if bytes.start_with?(prefix)
  end
  return 'image/webp' if bytes.bytesize >= 12 &&
                         bytes.byteslice(0, 4) == 'RIFF'.b &&
                         bytes.byteslice(8, 4) == 'WEBP'.b

  nil
end

.read_as_text(path) ⇒ String

Read path as plain UTF-8 text through the Extractor registry: unrecognised-but-textual passes through verbatim; with pikuri-pdf registered, PDFs get "--- Page N ---" markers (a scanned-image PDF comes back empty — a deliberate silent skip callers detect by length). Refusals raise (callers are internal pikuri code, not an LLM tool).

Parameters:

  • path (Pathname) —

    file to read.

Returns:

  • (String) —

    UTF-8 text; may be empty (empty file, scanned PDF).

Raises:

  • (ArgumentError) —

    if path isn't a Pathname, is a directory, an image, or opaque binary (the last mapped from Extractor::Unsupported).

  • (Errno::ENOENT) —

    if path doesn't exist.

  • (RuntimeError) —

    on an extraction failure (mapped from Extractor::Error, path included).



115
116
117
118
119
120
121
122
 
# File 'lib/pikuri/file_type.rb', line 115

def read_as_text(path)
  mime = guard_extractable(path)
  path.open('rb') { |io| Extractor.extract(io, content_type: mime) }
rescue Extractor::Unsupported
  raise ArgumentError, "#{path} appears to be binary; cannot extract as text"
rescue Extractor::Error => e
  raise "Cannot extract text from #{path}: #{e.message}"
end

.read_as_text_paged(path, offset: 1, limit: Extractor::PAGE_DEFAULT_LIMIT, max_bytes: Extractor::PAGE_MAX_BYTES, max_line_length: Extractor::PAGE_MAX_LINE_LENGTH) ⇒ Extractor::Page

Extract path into a windowed Extractor::Page (lines from offset up to limit, byte-capped, long lines truncated). Same routing and refusal contract as read_as_text; windowing semantics are Extractor.extract_paged's.

Parameters:

  • path (Pathname) —

    file to read.

  • offset (Integer) (defaults to: 1) —

    1-indexed first line; caller validates >= 1.

  • limit (Integer) (defaults to: Extractor::PAGE_DEFAULT_LIMIT) —

    max lines to collect; caller validates >= 1.

  • max_bytes (Integer) (defaults to: Extractor::PAGE_MAX_BYTES) —

    hard byte cap on collected content.

  • max_line_length (Integer) (defaults to: Extractor::PAGE_MAX_LINE_LENGTH) —

    per-line truncation threshold.

Returns:

Raises:

  • (ArgumentError) —

    if path isn't a Pathname, is a directory, an image, or binary.

  • (Errno::ENOENT) —

    if path doesn't exist.

  • (RuntimeError) —

    on an extraction failure.



139
140
141
142
143
144
145
146
147
148
149
150
151
 
# File 'lib/pikuri/file_type.rb', line 139

def read_as_text_paged(path, offset: 1, limit: Extractor::PAGE_DEFAULT_LIMIT,
                       max_bytes: Extractor::PAGE_MAX_BYTES,
                       max_line_length: Extractor::PAGE_MAX_LINE_LENGTH)
  mime = guard_extractable(path)
  path.open('rb') do |io|
    Extractor.extract_paged(io, content_type: mime, offset: offset, limit: limit,
                                max_bytes: max_bytes, max_line_length: max_line_length)
  end
rescue Extractor::Unsupported
  raise ArgumentError, "#{path} appears to be binary; cannot extract as text"
rescue Extractor::Error => e
  raise "Cannot extract text from #{path}: #{e.message}"
end