Module: Labimotion::FileExtractor

Defined in:
lib/labimotion/libs/file_extractor.rb

Overview

FileExtractor turns an uploaded file (base64-encoded raw bytes + filename) into plain text usable as LLM context for the AI dataset-template feature.

It NEVER raises into the caller. Any failure — missing gem, unsupported or oversized or corrupt file, parse error, timeout, out-of-memory — returns either '' or a short bracketed marker describing why, so the request keeps working with filename-only context.

Constant Summary collapse

TEXT_EXTS = 
%w[txt csv tsv md markdown json xml yml yaml].freeze
BINARY_EXTS = 
%w[pdf xlsx xlsm xls].freeze
ALLOWED_EXTS = 
(TEXT_EXTS + BINARY_EXTS).freeze
MAX_DECODED_BYTES =

reject decoded payloads larger than this

15 * 1024 * 1024
MAX_UNCOMPRESSED =

xlsx zip-bomb guard (sum of entry sizes)

80 * 1024 * 1024
MAX_OUT_CHARS =

per-file extracted-text cap

8000
MAX_PDF_PAGES = 
50
MAX_SHEETS = 
5
MAX_ROWS = 
100
MAX_COLS = 
30
MAX_CELL = 
200
EXTRACT_TIMEOUT =

best-effort (cannot interrupt C-ext parsing)

20
ZIP_MAGIC =

Magic bytes — forced to binary so start_with? never hits an Encoding::CompatibilityError against the ASCII-8BIT decoded bytes.

"PK\x03\x04".b
OLE2_MAGIC = 
"\xD0\xCF\x11\xE0".b
PDF_MAGIC = 
'%PDF'.b

Class Method Summary collapse

Class Method Details

.extract(filename, content_base64) ⇒ String

Returns extracted text, a '[...]' marker, or ''.

Parameters:

  • filename (String) —

    used for the extension whitelist + dispatch

  • content_base64 (String) —

    base64 of the raw file bytes (data-URL prefix tolerated)

Returns:

  • (String) —

    extracted text, a '[...]' marker, or ''



54
55
56
 
# File 'lib/labimotion/libs/file_extractor.rb', line 54

def extract(filename, content_base64)
  extract_bytes(filename, decode(content_base64))
end

.extract_bytes(filename, bytes) ⇒ String

Same as #extract but for already-decoded RAW bytes (e.g. Attachment#read_file). Keeps every guard #extract has: extension whitelist, empty check, MAX_DECODED_BYTES cap, the extraction Timeout, output truncation and the four rescue clauses. NEVER raises into the caller.

Parameters:

  • filename (String) —

    used for the extension whitelist + dispatch

  • bytes (String, nil) —

    raw file bytes (any encoding; treated as binary)

Returns:

  • (String) —

    extracted text, a '[...]' marker, or ''



66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/labimotion/libs/file_extractor.rb', line 66

def extract_bytes(filename, bytes)
  ext = File.extname(filename.to_s).downcase.delete('.')
  return '' unless ALLOWED_EXTS.include?(ext)

  return '' if bytes.nil? || bytes.empty?
  return '[file too large to extract]' if bytes.bytesize > MAX_DECODED_BYTES

  truncate(Timeout.timeout(EXTRACT_TIMEOUT) { dispatch(ext, bytes) })
rescue Timeout::Error
  '[extraction timed out]'
rescue NoMemoryError, SystemStackError => e
  log(e)
  '[file too complex to extract]'
rescue StandardError => e
  log(e)
  ''
end