Class: Rouge::Lexers::XQL

Inherits:
RegexLexer
  • Object
show all
Defined in:
lib/rouge/lexers/xql.rb,
lib/rouge/lexers/xql/builtins.rb

Overview

Lexer for XQL, the Cortex XSIAM/XDR query language.

Covers both dialects, which share one expression language:

* pipeline queries    `config timeframe = 30d | dataset = x | filter a ~= "b"`
* XIF rule files      `[INGEST:vendor="x", ...]` / `[RULE: NAME]` / `[MODEL: dataset="x"]`

Three properties of the language drive most of the rule ordering:

* inside `"..."` the ONLY escape is `\"` -- `\d`, `\s` and `\\` are ordinary
characters, which is why regexes are written raw. `"""..."""` is different: it
takes the JSON escape set, and is the only string form that may span newlines.
* `-` binds greedily into a number, so `a-1` is two tokens: `a` and `-1`.
* the language resolves ambiguity by longest match where Rouge takes the first
match, so `4625abc` is a single identifier and the number rules need a trailing
`(?!#{WORD_CHAR})` guard or they would split it.

Constant Summary collapse

WORD_CHAR =

An identifier is one or more of: digit, ASCII letter, underscore, or a character in [\u0080-\u201B] / [\u201E-\uFFFE] -- note the hole, which excludes the smart quotes U+201C and U+201D. Backtick-quoted names are handled by their own rule.

/[0-9A-Za-z_\u0080-\u201B\u201E-\uFFFE]/.freeze
QUOTED =

A double-quoted string, which cannot span a newline.

/"(?:\\"|[^"\r\n])*"/.freeze
CLOSING_QUOTE_AHEAD =

\" is the only escape, and \\ is NOT one -- so "\\" is ambiguous: the last two characters are either an escaped quote (leaving the string open) or a literal backslash and the closing quote. The language resolves this by longest valid token, i.e. \" is an escape only when a closing quote is still reachable on the same line. This lookahead, applied at the opening quote and at each \", reproduces that. Regexes ending in a backslash depend on it.

/(?=(?:\\"|[^"\r\n])*")/.freeze

Class Method Summary collapse

Class Method Details

.clause_keywords ⇒ Object

Words that modify a stage rather than opening one.



55
56
57
58
59
 
# File 'lib/rouge/lexers/xql.rb', line 55

def self.clause_keywords
  @clause_keywords ||= Set.new %w(
    as by asc desc for span bins suffix timeshift timezone
  )
end

.commands ⇒ Object

Stage commands -- the word that opens a query or follows a pipe. The last six are valid only in XIF rule files.



40
41
42
43
44
45
46
47
48
 
# File 'lib/rouge/lexers/xql.rb', line 40

def self.commands
  @commands ||= Set.new %w(
    aihelper search preset call call_macro dataset cold_dataset datamodel
    filter sort limit fields alter comp windowcomp target dedup bin union
    join pivot view arrayexpand replacenull config iploc transaction tag
    top getrole transpose hl highlight table graph
    drop storeset storeget switch case default
  )
end

.constants ⇒ Object 



67
68
69
 
# File 'lib/rouge/lexers/xql.rb', line 67

def self.constants
  @constants ||= Set.new %w(true false null)
end

.detect?(text) ⇒ Boolean

Returns:

  • (Boolean) 


93
94
95
96
97
98
 
# File 'lib/rouge/lexers/xql.rb', line 93

def self.detect?(text)
  # An XIF section header is unambiguous; nothing else opens a file this way.
  return true if text =~ /\A\s*(?:\/\*.*?\*\/|\/\/[^\n]*\n|\s)*\[\s*(?:INGEST|MODEL|RULE|CONST)\b/im

  false
end

.functions ⇒ Object 



10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
# File 'lib/rouge/lexers/xql/builtins.rb', line 10

def self.functions
  @functions ||= Set.new %w(
    abs acos add approx_count
    approx_quantiles approx_top array_all array_any
    array_length arrayconcat arraycreate arraydistinct
    arrayfilter arrayindex arrayindexof arraymap
    arraymerge arrayrange arraystring asin
    avg bitwise_and bitwise_or bitwise_sleft
    bitwise_sright bitwise_xor cbrt ceil
    coalesce concat convert_from_base_64 convert_to_base_64
    cos cosine_distance cot count
    count_distinct csc current_time date_floor
    divide earliest euclidean_distance exp
    extract_time extract_url_host extract_url_pub_suffix extract_url_registered_domain
    first first_value floor format_string
    format_timestamp greatest hierarchy_match if
    incidr incidr6 incidrlist int_to_ip
    ip_to_int is_ipv4 is_ipv6 is_known_private_ipv4
    is_known_private_ipv6 json_extract json_extract_array json_extract_scalar
    json_extract_scalar_array json_path_extract lag last
    last_value latest least len
    list ln log log10
    lowercase ltrim max md5
    median min mod multiply
    object_create object_merge parse_epoch parse_timestamp
    pow rand range_bucket rank
    regextract replace replex round
    row_number rtrim safe_add safe_divide
    safe_ip_to_int safe_multiply safe_negate safe_subtract
    sec sha1 sha256 sha512
    sign sin split sqrt
    stddev_population stddev_sample string_count subtract
    sum tan time_frame_end timestamp_diff
    timestamp_seconds to_boolean to_epoch to_float
    to_integer to_json_string to_number to_string
    to_timestamp trim trunc uppercase
    values var wildcard_match
  )
end

.join_types ⇒ Object

Only consulted directly before a (, where these are join types (join type = inner (...)) rather than calls. None is a builtin function.



73
74
75
 
# File 'lib/rouge/lexers/xql.rb', line 73

def self.join_types
  @join_types ||= Set.new %w(inner left right outer full cross)
end

.operator_words ⇒ Object 



61
62
63
64
65
 
# File 'lib/rouge/lexers/xql.rb', line 61

def self.operator_words
  @operator_words ||= Set.new %w(
    and or not in contains like between incidr incidr6
  )
end