Module: Synthra::Parser::Tokens

Included in:
Lexer, Parser
Defined in:
lib/synthra/parser/tokens.rb

Overview

Token type constants for the DSL lexer

This module defines all the token types that the lexer can produce. Each token type is a symbol that identifies the kind of syntax element.

Token types are organized into categories:

  • Structural: Schema and field names, colons, question marks
  • Types: Type names, Ref keyword
  • Delimiters: Parentheses, commas, dots
  • Values: Numbers, strings, identifiers, ranges, percentages
  • Keywords: if, true, false
  • Behaviors: @ prefixed behavior declarations
  • Whitespace/Structure: Newlines, indentation changes
  • Special: EOF, symbols

Examples:

Check if a token is a type

token.type == Tokens::TYPE

Check if token type is valid

Tokens::ALL.include?(:type)  # => true

Constant Summary collapse

SCHEMA_NAME =

Schema name at the top level (e.g., "User:") Value: String containing the schema name without the colon

Examples:

"User:" => Token(:schema_name, "User")

:schema_name
FIELD_NAME =

Field name within a schema (e.g., "name:", "email?:") Value: String containing the field name (may include ? for optional)

Examples:

"email?:" => Token(:field_name, "email?")

:field_name
COLON =

Colon separator Value: ":" literal

Examples:

":" => Token(:colon, ":")

:colon
QUESTION =

Question mark for optional/nullable types Value: "?" literal

Examples:

"?" => Token(:question, "?")

:question
TYPE =

Built-in or custom type name (uuid, text, number, etc.) Value: String containing the type name

Examples:

"uuid" => Token(:type, "uuid")

:type
REF =

Reference keyword for cross-schema references Value: "Ref" literal

Examples:

"Ref" => Token(:ref, "Ref")

:ref
LPAREN =

Left parenthesis for type arguments Value: "(" literal

Examples:

"(" => Token(:lparen, "(")

:lparen
RPAREN =

Right parenthesis for type arguments Value: ")" literal

Examples:

")" => Token(:rparen, ")")

:rparen
COMMA =

Comma separator in argument lists Value: "," literal

Examples:

"," => Token(:comma, ",")

:comma
DOT =

Dot separator in references (Schema.field) Value: "." literal

Examples:

"." => Token(:dot, ".")

:dot
MINUS =

Minus sign for negative numbers Value: "-" literal

Examples:

"-" => Token(:minus, "-")

:minus
STAR =

Arithmetic operators (used inside formula(...) expressions)

Examples:

"" => Token(:star, "")

:star
SLASH =

Examples:

"/" => Token(:slash, "/")

:slash
PLUS =

Examples:

"+" => Token(:plus, "+")

:plus
NUMBER =

Numeric literal (integer or float) Value: Integer or Float

Examples:

"123" => Token(:number, 123)

"3.14" => Token(:number, 3.14)

:number
STRING =

String literal (single or double quoted) Value: String (quotes stripped)

Examples:

'"hello"' => Token(:string, "hello")

:string
IDENTIFIER =

Identifier (unquoted name, used in enums, etc.) Value: String

Examples:

"pending" => Token(:identifier, "pending")

:identifier
RANGE =

Range literal (min..max) Value: Range object

Examples:

"18..80" => Token(:range, 18..80)

:range
PERCENT =

Percentage literal Value: Integer (percent without %)

Examples:

"50%" => Token(:percent, 50)

:percent
IF =

Conditional keyword for field conditions Value: "if" literal

Examples:

"if" => Token(:if, "if")

:if
TRUE_KW =

Boolean true keyword Value: true

Examples:

"true" => Token(:true, true)

:true
FALSE_KW =

Boolean false keyword Value: false

Examples:

"false" => Token(:false, false)

:false
SCHEMA_KEYWORD =

Schema definition keyword Value: "schema" literal

Examples:

"schema User:" => Token(:schema_keyword, "schema")

:schema_keyword
API_KEYWORD =

API definition keyword Value: "api" literal

Examples:

"api GET /users:" => Token(:api_keyword, "api")

:api_keyword
SCENARIO_KEYWORD =

Scenario definition keyword Value: "scenario" literal

Examples:

"scenario Success:" => Token(:scenario_keyword, "scenario")

:scenario_keyword
HTTP_METHOD =

HTTP method token Value: Symbol (:get, :post, :put, :patch, :delete, :options, :head)

Examples:

"GET" => Token(:http_method, :get)

:http_method
PATH =

URL path token Value: String containing the path

Examples:

"/users/:id" => Token(:path, "/users/:id")

:path
SCHEMA_REF =

Schema reference token for explicit schema(:Name) syntax Value: String containing the schema name

Examples:

"schema(:User)" => Token(:schema_ref, "User")

:schema_ref
AT_BEHAVIOR =

Behavior annotation (@latency, @failure, etc.) Value: String containing behavior name without @

Examples:

"@latency" => Token(:at_behavior, "latency")

:at_behavior
NEWLINE =

Newline character Value: "\n" literal

Examples:

"\n" => Token(:newline, "\n")

:newline
INDENT =

Increase in indentation level Value: Integer (new indentation level)

Examples:

Indent from 0 to 2 => Token(:indent, 2)

:indent
DEDENT =

Decrease in indentation level Value: Integer (new indentation level)

Examples:

Dedent from 4 to 2 => Token(:dedent, 2)

:dedent
EOF =

End of file marker Value: nil

Examples:

End of input => Token(:eof, nil)

:eof
SYMBOL =

Ruby symbol literal (:function_name) Value: Symbol

Examples:

":compute" => Token(:symbol, :compute)

:symbol
ALL =

Array of all valid token types

Examples:

Validate token type

Tokens::ALL.include?(token.type)  # => true/false

Returns:

  • (Array<Symbol>) —

    all token type symbols

[
  SCHEMA_NAME, FIELD_NAME, COLON, QUESTION,
  TYPE, REF,
  LPAREN, RPAREN, COMMA, DOT, MINUS,
  NUMBER, STRING, IDENTIFIER, RANGE, PERCENT,
  IF, TRUE_KW, FALSE_KW,
  SCHEMA_KEYWORD, API_KEYWORD, SCENARIO_KEYWORD, HTTP_METHOD, PATH, SCHEMA_REF,
  AT_BEHAVIOR,
  NEWLINE, INDENT, DEDENT,
  EOF, SYMBOL
].freeze