Module: Pikuri::Code::Bash::Tokenizer

Defined in:: lib/pikuri/code/bash/tokenizer.rb

Overview

A deliberately narrow shell tokenizer: turns a command string into one word list per sequencer segment, or nil when it can't prove the command is such a simple chain. It makes no passivity judgement — that policy lives in PassiveCommandDetector, which classifies these word lists. Lexing (here) is split from classification (the passive detector) so a second consumer could reuse the parse.

Tokenizer.tokenize returns Array<Array<String>> — segments (cut on SEQUENCERS) of de-quoted, de-redirected words, an empty list marking a pure-comment no-op — or nil meaning "more than a simple chain" (a metacharacter, unsafe redirect, unbalanced quote, too many segments). Conservative by contract: when in doubt it returns nil so the caller delegates to a human; it never silently drops a dangerous byte.

allow_glob: is the one knob: off (default) an unquoted glob/brace/ tilde (GLOB_CHARS) rejects like any metacharacter; on, it is kept as a literal word byte and the authorization layer decides. This is sound because those chars only ever expand to words, never to an operator or a new command — so keeping them literal can't smuggle a +;+/+|+/+$()+ past the lexer. The detector flips it on only for the pure-allowlist glob fast path; see PassiveCommandDetector.

Why splitting on the sequencers is faithful

Tokenizer.split_segments cuts only on a genuine control operator, never one hidden inside a string: it tracks quote state (single and double) and won't split inside a quoted span, and the backslash stays forbidden, so nothing escapes a +;+/+|+/newline past it. Every other metacharacter stays forbidden per segment, so an arbitrary-target redirect (+ps | grep x > out+), a background & (+ls & pwd+) or |& trip the metacharacter check ⇒ nil. The lone exception is a write-nothing redirect fragment (SAFE_REDIRECT: 2>&1, 2>/dev/null …), consumed and dropped — fd duplication and /dev/null discards add no capability.

Quote removal

Tokenizer.scan_segment performs genuine shell quote-removal: the marks are dropped and the interior kept verbatim and literal, so a quoted | neither splits nor trips the gate (+echo 'a | b'+ is one echo). Single quotes decode unconditionally — the POSIX interior is purely literal, closed by the next '. Double quotes decode only when verifiably inert: POSIX keeps three characters live inside "..." — $ (expansion), the backtick (command substitution), \ (escaping) — so a span containing any DQUOTE_LIVE byte returns nil (delegate, never guess), and one containing none is decoded like a single-quoted span. This admits the common LLM shape grep "foo bar" f while keeping the failure mode "asks the human too often". An unbalanced quote → nil.

Genuine quote-removal (not a placeholder rewrite) is what makes the spelling-based classifiers in PassiveCommandDetector sound — the cross-cutting "why the de-quoted word must be the true word, and why an allowlist tolerates a shortcut a denylist doesn't" argument lives in pikuri-code/DESIGN.md (The soundness hinge: true de-quoted words).

The classic mis-parse — echo "a\"b; rm x", where a naive lexer closes the span at the escaped quote and exposes the ; — cannot happen: the scan bails on the \ before reaching the quote it escapes. (Tokenizer.split_segments's simpler tracker does close a span at an escaped quote, but any command where that matters carries a \ outside single quotes into some segment, which Tokenizer.scan_segment rejects — so the divergence can only produce nil, never a mis-parsed approval.)

Bash's fourth live character, ! (history expansion), is NOT gated inside the span: it fires only in interactive shells, and Pikuri::Code::Bash always spawns bash -c (histexpand off), so echo "done!" is literal in every shell this reaches. Outside quotes ! stays forbidden.

Constant Summary collapse

SHELL_METACHARACTERS = Any shell metacharacter or ASCII control byte that, outside a quoted span, means the segment is more than a single simple invocation ⇒ nil. scan_segment consults this per character only after handling quotes, whitespace, the SAFE_REDIRECT forms and GLOB_CHARS, so a safe redirect's +>+/+&+ and every quoted-argument byte never reach it. The backslash \ stays here (still forbidden): decoding it would reopen the parsing surface the quote handling avoids. The double quote is not listed — like ' it is handled by scan_segment's quote branches first. The glob/brace/tilde chars are not here either — they live in GLOB_CHARS, which scan_segment rejects by default but keeps as literal word bytes under allow_glob:. Note = is not here (+--color=never+); a leading VAR=val is caught because the first word then isn't a bare allowed binary.

/[;&|<>$`()\\!#\x00-\x1f]/

GLOB_CHARS = The shell expansion characters — pathname globbing (+*+ ? [...]), brace expansion (+a,b+), tilde expansion (+~+). Split out of SHELL_METACHARACTERS because they are the one metacharacter class that only ever expands to words (filenames/paths), never to an operator or a new command (bash tokenizes operators before expansion and never re-scans the result). scan_segment rejects them by default (⇒ nil, as before), but under allow_glob: true keeps them as literal word bytes so the caller can classify them where sound — see tokenize's allow_glob: and the PassiveCommandDetector glob fast path.

/[*?\[\]{}~]/

DQUOTE_LIVE = The three characters POSIX keeps live inside "..." — $ (expansion), the backtick (command substitution), \ (escaping, including of the closing "). scan_segment returns nil when any appears inside a double-quoted span; every other interior byte is literal. See the class header on why ! is not gated.

/[$`\\]/

SEQUENCERS = Shell control operators that merely sequence or pipe commands. split_segments cuts on these outside quotes only (a quoted +|+/+;+ is literal data). Alternation order matters: two-char +&&+/+||+ before single | so a || b splits into two segments, not three. A newline is a command terminator like ;; a literal newline would sit inside a quoted span, where it is treated as data.

/&&|\|\||;|\||\n/

MAX_CHAIN = Defensive ceiling on sequencer-joined segments — a sanity backstop against pathological input, set high enough to clear a real generated batch (a dpkg -S … || true probe per unit file across dozens of files). Beyond it the whole command returns nil.

SAFE_REDIRECT = A redirect fragment that writes nothing. scan_segment anchors this at a > (or &>) and, on a match, consumes and drops it (so the +>+/+&+ never reaches the SHELL_METACHARACTERS gate); a non-matching > is a real redirect ⇒ nil. Two safe families: fd duplication — [n]>&[m] (+2>&1+ merges stderr into stdout): pure in-process fd-table manipulation, never a file. discard to /dev/null — [n|&]>[>] /dev/null (append forms too). The target is pinned to exactly /dev/null by the (?=\s|\z) lookahead, so a look-alike (+2>/dev/nullx+) does not match ⇒ nil, as does any arbitrary-file redirect. \s* accepts glued or spaced. The leading [0-9]* is the fd scan_segment accumulated into the current word (the 2 of 2>&1); it trims those digits on a match.

%r{[0-9]*>&[0-9]+|(?:[0-9]|&)?>>?\s*/dev/null(?=\s|\z)}

Class Method Summary collapse

.tokenize(command, allow_glob: false) ⇒ Array<Array<String>>^?
Parse the command into one word list per SEQUENCERS segment, or nil if it can't be safely analyzed as such a chain.

Class Method Details

.tokenize(command, allow_glob: false) ⇒ `Array<Array<String>>`^?

Parse the command into one word list per SEQUENCERS segment, or nil if it can't be safely analyzed as such a chain.

Parameters:

command (String) —
the command (a bin/pikuri-* caller should strip any "$ " echo prefix first).
allow_glob (Boolean) (defaults to: false) —
when true, an unquoted GLOB_CHARS byte is kept as a literal word byte instead of rejecting the segment. Default false (a glob ⇒ nil, the strict behavior). A quoted glob char is literal either way — this flag only affects unquoted ones. The caller opts in only where a post-expansion glob is provably harmless (see the PassiveCommandDetector glob fast path).

Returns:

(Array<Array<String>>, nil) —
one de-redirected, de-quoted word list per segment (empty list = pure-comment no-op), or nil — more than MAX_CHAIN segments, an unbalanced quote, a DQUOTE_LIVE byte inside double quotes, an unquoted GLOB_CHARS byte (unless allow_glob:), or any segment empty (a dangling sequencer) or not a simple invocation.

# File 'lib/pikuri/code/bash/tokenizer.rb', line 159

def tokenize(command, allow_glob: false)
  segments = split_segments(command)
  return nil if segments.nil?

  words_per_segment = segments.map { |s| scan_segment(s, allow_glob) }
  return nil if words_per_segment.any?(&:nil?)

  words_per_segment
end