Module: Clacky::Utils::ParserManager

Defined in:
lib/clacky/utils/parser_manager.rb

Overview

Manages user-space parsers in ~/.clacky/parsers/.

On first use, default parser scripts are copied from the gem's default_parsers/ directory into ~/.clacky/parsers/. After that, the user-space version is always used — allowing the LLM to modify or extend parsers without touching the gem itself.

CLI interface contract (all parsers must follow):

ruby <parser>.rb <file_path>
stdout → extracted text (UTF-8)
stderr → error messages
exit 0 → success
exit 1 → failure

Constant Summary collapse

PARSERS_DIR = 
File.expand_path("~/.clacky/parsers").freeze
DEFAULT_PARSERS_DIR = 
File.expand_path("../default_parsers", __dir__).freeze
PARSER_FOR = 
{
  ".pdf"  => "pdf_parser.rb",
  ".doc"  => "doc_parser.rb",
  ".docx" => "docx_parser.rb",
  ".xlsx" => "xlsx_parser.py",
  ".xls"  => "xlsx_parser.py",
  ".pptx" => "pptx_parser.rb",
  ".ppt"  => "pptx_parser.rb",
  ".wps"  => "wps_parser.rb",
  ".et"   => "wps_parser.rb",
  ".dps"  => "wps_parser.rb",
}.freeze
INTERPRETER_FOR =

Map a parser script's extension to the interpreter that runs it. Lets PARSER_FOR point at scripts in any language (see extract_version).

{ ".rb" => RbConfig.ruby, ".py" => "python3" }.freeze
PYTHON_PARSER_LIBS =

Third-party libraries a given Python parser needs at runtime.

{ "xlsx_parser.py" => "openpyxl" }.freeze
PARSE_TIMEOUT =

Hard ceiling on how long a single parser subprocess may run. A runaway parser (huge/malformed file) is killed rather than hanging the caller forever and starving the machine.

60

Class Method Summary collapse

Class Method Details

.capture3_with_timeout(*cmd, timeout:) ⇒ Object

Run a subprocess with a hard timeout. On timeout the whole process GROUP is killed (TERM, 2s grace, then KILL) so grandchildren spawned by the parser die too. Mirrors mcp/stdio_transport.rb's kill sequence.

Returns [stdout, stderr, status] — status is a Process::Status on normal exit, or the symbol :timeout when the subprocess was killed.



195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
 
# File 'lib/clacky/utils/parser_manager.rb', line 195

def self.capture3_with_timeout(*cmd, timeout:)
  stdin, stdout, stderr, wait_thr = Open3.popen3(*cmd, pgroup: true)
  stdin.close
  pgid = Process.getpgid(wait_thr.pid)

  out_thr = Thread.new { stdout.read }
  err_thr = Thread.new { stderr.read }

  if wait_thr.join(timeout)
    [out_thr.value, err_thr.value, wait_thr.value]
  else
    kill_process_group(pgid)
    out_thr.kill
    err_thr.kill
    [nil, "timed out", :timeout]
  end
ensure
  [stdout, stderr].each { |io| io&.close rescue nil }
end

.ensure_python_deps(script) ⇒ Object

Ensure Python 3 and the libs a Python parser needs are present, installing on demand. Returns nil on success, or an error string.

If python3 is missing, returns an error instructing the caller (the LLM via terminal tool) to run install_system_deps.sh --clt-only — we do NOT run it here because it blocks for 100+ seconds (CLT download). If python3 exists, probe + install the missing lib via pip --user.



242
243
244
245
246
247
248
249
250
251
252
253
 
# File 'lib/clacky/utils/parser_manager.rb', line 242

def self.ensure_python_deps(script)
  lib = PYTHON_PARSER_LIBS[script]

  unless python3_available?
    return "Python 3 is required to parse this file. " \
           "Run: bash ~/.clacky/scripts/install_system_deps.sh --clt-only\n" \
           "Then retry."
  end

  return nil if lib.nil? || python_lib_present?(lib)
  pip_install(lib) ? nil : "Failed to install #{lib} (required to parse this file)."
end

.extract_version(path) ⇒ Object

Read the VERSION marker from a parser script (e.g. "# VERSION: 2"). Works for any script language that uses # for comments (Ruby, Python, shell). Returns Integer or nil.



114
115
116
117
118
119
120
121
122
123
124
125
126
 
# File 'lib/clacky/utils/parser_manager.rb', line 114

def self.extract_version(path)
  return nil unless File.exist?(path)
  # Only scan the first 40 lines — the marker lives in the header.
  File.foreach(path).with_index do |line, i|
    break if i >= 40
    if (m = line.match(/^\s*#\s*VERSION:\s*(\d+)/i))
      return m[1].to_i
    end
  end
  nil
rescue StandardError
  nil
end

.interpreter_for(script) ⇒ Object

Map a parser script to its interpreter (Ruby, Python, ...).



185
186
187
 
# File 'lib/clacky/utils/parser_manager.rb', line 185

def self.interpreter_for(script)
  INTERPRETER_FOR[File.extname(script)] || RbConfig.ruby
end

.kill_process_group(pgid) ⇒ Object

Terminate a process group: TERM, wait up to 2s, then KILL.



216
217
218
219
220
221
222
223
224
225
226
 
# File 'lib/clacky/utils/parser_manager.rb', line 216

def self.kill_process_group(pgid)
  Process.kill("TERM", -pgid)
rescue Errno::ESRCH, Errno::EPERM
else
  deadline = Time.now + 2
  sleep 0.05 while process_group_alive?(pgid) && Time.now < deadline
  begin
    Process.kill("KILL", -pgid) if process_group_alive?(pgid)
  rescue Errno::ESRCH, Errno::EPERM
  end
end

.parse(file_path) ⇒ Hash

Run the appropriate parser for the given file path.

Parameters:

  • file_path (String) —

    path to the file to parse

Returns:

  • (Hash) —

    { success: bool, text: String, error: String, parser_path: String }



132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/clacky/utils/parser_manager.rb', line 132

def self.parse(file_path)
  ext = File.extname(file_path.to_s).downcase
  script = PARSER_FOR[ext]

  unless script
    return { success: false, text: nil,
             error: "No parser available for #{ext} files",
             parser_path: nil }
  end

  parser_path = File.join(PARSERS_DIR, script)

  unless File.exist?(parser_path)
    return { success: false, text: nil,
             error: "Parser not found: #{parser_path}",
             parser_path: parser_path }
  end

  interpreter = interpreter_for(script)

  # Python parsers need Python + their libs present before parsing.
  # ensure_python_deps returns an error string if python3 is missing or
  # a lib can't be installed — the caller surfaces it as parse_error.
  if interpreter == "python3"
    dep_error = ensure_python_deps(script)
    return { success: false, text: nil, error: dep_error, parser_path: parser_path } if dep_error
  end

  raw_stdout, raw_stderr, status =
    capture3_with_timeout(interpreter, parser_path, file_path, timeout: PARSE_TIMEOUT)

  # capture3 returns ASCII-8BIT across the subprocess boundary on Ruby 2.6+.
  # Normalise both streams to UTF-8 immediately so all downstream code is clean.
  stdout = Clacky::Utils::Encoding.to_utf8(raw_stdout.to_s)
  stderr = Clacky::Utils::Encoding.to_utf8(raw_stderr.to_s)

  # Filter out Ruby/Bundler version warnings that pollute stderr
  clean_stderr = stderr.lines.reject { |l| l.match?(/warning:|already initialized constant/) }.join.strip

  if status == :timeout
    { success: false, text: nil,
      error: "Parser timed out after #{PARSE_TIMEOUT}s (file too large or malformed)",
      parser_path: parser_path }
  elsif status.success? && stdout.strip.length > 0
    { success: true, text: stdout.strip, error: nil, parser_path: parser_path }
  else
    { success: false, text: nil,
      error: clean_stderr.empty? ? "Parser exited with code #{status.exitstatus}" : clean_stderr,
      parser_path: parser_path }
  end
end

.parser_path_for(ext) ⇒ Object

Returns the path to a parser script for a given extension. Used by agent to tell LLM where to find/modify the parser.



269
270
271
272
273
 
# File 'lib/clacky/utils/parser_manager.rb', line 269

def self.parser_path_for(ext)
  script = PARSER_FOR[ext.downcase]
  return nil unless script
  File.join(PARSERS_DIR, script)
end

.pip_install(lib) ⇒ Object 



263
264
265
 
# File 'lib/clacky/utils/parser_manager.rb', line 263

def self.pip_install(lib)
  system("python3", "-m", "pip", "install", "--user", lib, out: File::NULL, err: File::NULL)
end

.process_group_alive?(pgid) ⇒ Boolean

Returns:

  • (Boolean) 


228
229
230
231
232
233
 
# File 'lib/clacky/utils/parser_manager.rb', line 228

def self.process_group_alive?(pgid)
  Process.kill(0, -pgid)
  true
rescue Errno::ESRCH, Errno::EPERM
  false
end

.python3_available? ⇒ Boolean

Returns:

  • (Boolean) 


255
256
257
 
# File 'lib/clacky/utils/parser_manager.rb', line 255

def self.python3_available?
  system("python3", "--version", out: File::NULL, err: File::NULL)
end

.python_lib_present?(lib) ⇒ Boolean

Returns:

  • (Boolean) 


259
260
261
 
# File 'lib/clacky/utils/parser_manager.rb', line 259

def self.python_lib_present?(lib)
  system("python3", "-c", "import #{lib}", out: File::NULL, err: File::NULL)
end

.setup! ⇒ Object

Called at Agent startup (idempotent — safe to run every time).

Copies every file from default_parsers/ (not just the entry-point .rb scripts listed in PARSER_FOR). A parser may ship companion helper scripts — e.g. pdf_parser_ocr.py sits next to pdf_parser.rb and is invoked by relative path — so those helpers must be distributed too.

Version upgrade policy:

Each bundled parser declares `VERSION: <n>` in a header comment
(works for Ruby `# VERSION: 2` and Python `# VERSION: 2` alike,
scanned in the first 40 lines of the file).

On startup, per-file:
- If the file does NOT exist in ~/.clacky/parsers/ → copy it.
- If it exists:
    * bundled has no VERSION → never touch (bundled file
      is opting out of managed upgrades).
    * installed has no VERSION → treat it as legacy v0 and
      upgrade (lenient mode — covers users who installed before
      the VERSION scheme existed). The old file is backed up.
    * both have VERSION, bundled > installed → upgrade, backing
      up the old copy as `<script>.v<old>.bak`.
    * bundled ≤ installed → leave the user's copy alone
      (preserves LLM/user modifications).

Backups live alongside the parser so the user can inspect
their own edits after an upgrade. They are never removed
automatically.


77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'lib/clacky/utils/parser_manager.rb', line 77

def self.setup!
  FileUtils.mkdir_p(PARSERS_DIR)

  Dir.glob(File.join(DEFAULT_PARSERS_DIR, "**", "*")).each do |src|
    next unless File.file?(src)
    basename = File.basename(src)
    next if basename.start_with?(".") || basename.end_with?(".bak")

    rel  = src.sub(/^#{Regexp.escape(DEFAULT_PARSERS_DIR)}\/?/, "")
    dest = File.join(PARSERS_DIR, rel)

    if !File.exist?(dest)
      FileUtils.mkdir_p(File.dirname(dest))
      FileUtils.cp(src, dest)
      # Preserve executable bit so sibling scripts can be run directly.
      FileUtils.chmod(File.stat(src).mode, dest)
      next
    end

    bundled_version = extract_version(src)
    # Bundled file opts out of managed upgrades — never touch user copy.
    next unless bundled_version

    installed_version = extract_version(dest) || 0

    if bundled_version > installed_version
      backup = "#{dest}.v#{installed_version}.bak"
      FileUtils.cp(dest, backup) unless File.exist?(backup)
      FileUtils.cp(src, dest)
      FileUtils.chmod(File.stat(src).mode, dest)
    end
  end
end