rouge-xql

A Rouge lexer for XQL, the Cortex XSIAM/XDR query language — covering both of its dialects:

  • XQL Queries — config timeframe = 30d | dataset = xdr_data | filter a ~= "b"
  • Parsing & Modeling rules — [INGEST:...], [RULE: NAME], [MODEL:...], [CONST]

Both register under one tag, xql, because they share the entire expression language and real-world tooling feeds both through the same highlighter.

Install

# Gemfile
gem "rouge-xql"

require "rouge-xql"

Rouge::Lexer.find("xql")             # => Rouge::Lexers::XQL
Rouge.highlight(source, "xql", "html")

Requiring the gem is all that is needed: Rouge's tag DSL registers the class as a side effect of defining it. xif and xsiam work as aliases, and *.xql / *.xif are recognised by Rouge::Lexer.guess.

Pair it with any Rouge theme — the lexer emits only standard token types, so stock stylesheets (including a plain rougify style github dump) style it correctly with no extra CSS.

Notes on the language

XQL has a few properties that highlighters routinely get wrong. This lexer follows what the language actually accepts, and each case is covered by a test:

  • Inside "..." the only escape is \". \d, \s and \\ are ordinary characters, which is why regexes are written raw. """...""" is different — it takes the JSON escape set and is the only string form that may span a newline.
  • Because \\ is not an escape, "...\\" is ambiguous. It resolves by longest valid token: \" counts as an escape only when a closing quote is still reachable on the line. Get this wrong and a regex ending in a backslash swallows the rest of the line.
  • - binds greedily into a number: a-1 is Name(a) Num(-1), but fields -foo is Keyword(fields) Operator(-) Name(foo).
  • xdm.a.b.c is a single token. ENUM.X and XDM_CONST.X are three, but are highlighted as one unit since that is what they mean.
  • Backticks are an identifier quote, not a string — that is how a reserved word becomes a field name (`target`).
  • [], {} and -> are single tokens; a lone {` or `} is invalid.
  • There are no hex literals, no exponent floats, no => or <>, and no single-quoted strings. All are common in third-party XQL highlighting definitions and none is real.
  • Every keyword is case-insensitive; OR, AND and IN are commonly uppercase in the wild.

Two spellings are deliberately not treated as keywords: colddataset and callmacro. Some tooling lists them, but the language accepts only cold_dataset and call_macro, so highlighting them as commands would hide a real syntax error.

Field names are not baked in. They are per-deployment schema rather than language vocabulary, and the language itself draws no distinction between a field and any other identifier. Fields are highlighted structurally instead: xdm.* paths, XDM_CONST.* / ENUM.* constants, $refs and backtick-quoted names each get their own token type.

Known limitation

A string directly after ~= / !~= is lexed as a regex (Str::Regex), with its escapes picked out. The regex argument of regextract(field, "...") and replex is not — finding it needs paren-depth tracking through nested calls like regextract(to_string(_time), "..."). Those render as ordinary strings.

Development

bundle install
bundle exec rake                                   # specs
TEST=spec/lexers/xql_spec.rb bundle exec rake spec # one file

The suite covers the token types, the string and comment edge cases above, filename and mimetype guessing, and asserts that both the demo and the visual sample lex with zero Error tokens and round-trip byte for byte.

spec/visual/samples/xql is a single file exercising both dialects and most of the language; it is the quickest way to eyeball a change. Every identifier in it is invented.

Licence

MIT. Not affiliated with, or endorsed by, Palo Alto Networks.