Module: Ibex::Runtime::GeneratedLexer

Defined in:
lib/json5/generated_parser.rb

Overview

Runtime contract mixed into parser classes that declare a lexer. rubocop:disable Metrics/ModuleLength -- matching, actions, state, and positions share one session invariant.

Constant Summary collapse

NO_EMISSION =

Signature:

  • Object

Object.new.freeze
EMPTY_GREEN_TRIVIA =

Signature:

  • Array[CST::GreenTrivia]

empty_green_trivia.freeze

Instance Method Summary collapse

Instance Method Details

#lex(source, file: "(input)") ⇒ Object

Reset the generated lexer to the beginning of an input.

RBS:

  • (String | IO | Fiber source, ?file: String) -> self



8949
8950
8951
8952
8953
8954
8955
8956
8957
8958
8959
8960
8961
8962
8963
8964
 
# File 'lib/json5/generated_parser.rb', line 8949

def lex(source, file: "(input)")
  @lexer_input = LexerInput.new(source)
  @lexer_file = file
  @lexer_states = ["INITIAL"]
  @lexer_line = 1
  @lexer_column = 1
  @lexer_byte_column = 1
  @lexer_byte_offset = 0
  @lexer_lexeme = nil
  @lexer_emission = NO_EMISSION
  @lexer_skip_requested = false
  @lexer_pending_green_trivia = []
  configure_lexer_cst
  @lexer_has_token = false
  self
end

#lexer_state ⇒ Object

Return the current named lexer state.

RBS:

  • () -> Symbol



9010
9011
9012
 
# File 'lib/json5/generated_parser.rb', line 9010

def lexer_state
  (@lexer_states || ["INITIAL"]).last.to_sym
end

#lexer_state=(state) ⇒ Object

Replace the current named lexer state.

RBS:

  • (Symbol | String state) -> Symbol



9016
9017
9018
9019
9020
9021
 
# File 'lib/json5/generated_parser.rb', line 9016

def lexer_state=(state)
  name = validate_lexer_state(state)
  states = @lexer_states ||= ["INITIAL"]
  states[-1] = name
  name.to_sym
end

#next_token ⇒ Object

Pull one token using longest match and declaration-order tie breaking.

RBS:

  • () -> [Object?, Object?, Hash[Symbol, Object?]]

Raises:



9025
9026
9027
9028
9029
9030
9031
9032
9033
9034
9035
9036
9037
9038
9039
9040
9041
9042
9043
9044
9045
 
# File 'lib/json5/generated_parser.rb', line 9025

def next_token
  input = @lexer_input
  raise ParseError, "(lexer):1:1: call parse or lex before next_token" unless input

  loop do
    unless ensure_lexer_data?(input)
      location = lexer_zero_width_location
      location[:ibex_lexer_start_state] = lexer_state
      location = attach_cst_trivia(location) if cst_enabled?
      return [nil, nil, location.freeze]
    end

    rule, lexeme = select_lexer_match(input)
    raise_lexer_no_match(input) unless rule && lexeme

    location = consume_lexer_match(input, lexeme)
    location[:ibex_lexer_start_state] = lexer_state
    emitted = apply_lexer_rule(rule, lexeme, location)
    return emitted if emitted
  end
end

#parse(source, file: "(input)") ⇒ Object

Lex and semantically parse one String, IO, or Fiber source. Parser and generated lexer actions execute. Loading the generated file may also execute user sections; this trusted application path is not a sandbox.

RBS:

  • (String | IO | Fiber source, ?file: String) -> Object?



8971
8972
8973
8974
8975
8976
8977
8978
8979
8980
 
# File 'lib/json5/generated_parser.rb', line 8971

def parse(source, file: "(input)")
  lex(source, file: file)
  parser = self #: Parser
  parser.do_parse
rescue ParseError => e
  raise unless parser_tables[:cst].is_a?(Hash)

  parser = self #: Parser
  parser.__send__(:cst_lexical_failure, e)
end

#parse_syntax(source, file: "(input)") ⇒ Object

Parse without executing parser production actions. Generated lexer actions still run because they define tokenization and lexer state. Loading user sections and running lexer actions are trusted application boundaries, so this method is not a no-user-code sandbox.

RBS:

  • (String source, ?file: String) -> CST::SyntaxResult



9004
9005
9006
 
# File 'lib/json5/generated_parser.rb', line 9004

def parse_syntax(source, file: "(input)")
  parse_syntax_with_cache(CST::SourceText.new(source, file: file), CST::NodeCache.new)
end

#parse_with_syntax(source, file: "(input)") ⇒ Object

Parse one generated-lexer source and return its semantic and syntax results. Parser and generated lexer actions execute. This trusted application path is not a sandbox.

RBS:

  • (String | IO | Fiber source, ?file: String) -> CST::ParseResult



8986
8987
8988
8989
8990
8991
8992
8993
8994
8995
8996
8997
 
# File 'lib/json5/generated_parser.rb', line 8986

def parse_with_syntax(source, file: "(input)")
  lex(source, file: file)
  parser = self #: Parser
  value = parser.do_parse
  parser.__send__(:syntax_parse_result, value)
rescue ParseError => e
  raise unless parser_tables[:cst].is_a?(Hash)

  parser = self #: Parser
  value = parser.__send__(:cst_lexical_failure, e)
  parser.__send__(:syntax_parse_result, value)
end