Class: Ibex::Runtime::Parser

Inherits:
Object
  • Object
show all
Includes:
Observation, ParserSyncRecovery
Defined in:
lib/json5/generated_parser.rb

Overview

Drives a table-defined LR parser without native extensions.

Subclasses provide .parser_tables, returning :tokens, :token_names, :actions, :gotos, and :productions, with optional :default_actions, :error_messages, and :recovery_sync_tokens. Actions are represented by [:shift, state], [:reduce, production], [:accept], or [:error]. Format-v2 and newer generated production entries mark their five-argument semantic methods with location_action: true. Format-v3 and newer composed actions additionally use composition_action: true for the six-argument contract carrying the lookahead location. Format-v4 location-free generated methods use values_action: true for a one-argument values contract. Format-v5 additionally marks proven-safe zero-to-four-value methods with positional_action: true. V1 and unmarked application actions retain the historical two-argument contract. Markers are honored only for the generated _ibex_action_N Symbol shape, never for callables.

Constant Summary collapse

ParseError =

Signature:

  • singleton(Ibex::Runtime::ParseError)

Ibex::Runtime::ParseError
EOF_TOKEN =

Signature:

  • Integer

0
ERROR_TOKEN =

Signature:

  • Integer

1
GENERATED_ACTION_NAME =

Signature:

  • Regexp

/\A_ibex_action_\d+\z/
NO_LOOKAHEAD =

Signature:

  • Object

Object.new.freeze
RECOVERY_SHIFTS =

Signature:

  • Integer

3
EMPTY_ROW =

Signature:

  • Hash[Integer, runtime_value]

empty_row.freeze
EMPTY_LOCATION_NAMES =

Signature:

  • Hash[Symbol, Integer]

empty_location_names.freeze
EMPTY_LOCATIONS =

Signature:

  • Array[runtime_value]

empty_locations.freeze
EMPTY_GREEN_TRIVIA =

Signature:

  • Array[CST::GreenTrivia]

empty_green_trivia.freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Observation

#observe, #unobserve

Constructor Details

#initialize(resource_limits: ResourceLimits.new) ⇒ Parser

Returns a new instance of Parser.

RBS:

  • (?resource_limits: ResourceLimits) -> void



6043
6044
6045
6046
 
# File 'lib/json5/generated_parser.rb', line 6043

def initialize(resource_limits: ResourceLimits.new)
  validate_resource_limits!(resource_limits)
  initialize_runtime_state(resource_limits, preserve_existing: false)
end

Instance Attribute Details

#incremental_reused_descendants ⇒ Object (readonly)

Signature:

  • Integer



5997
5998
5999
 
# File 'lib/json5/generated_parser.rb', line 5997

def incremental_reused_descendants
  @incremental_reused_descendants
end

#syntax_parse_memo ⇒ Object (readonly)

RBS:

  • @yydebug: bool

  • @yydebug_output: IO

  • @source: runtime_value

  • @state_stack: Array[Integer]

  • @value_stack: Array[runtime_value]

  • @vstack: Array[runtime_value]

  • @racc_vstack: Array[runtime_value]

  • @location_stack: Array[runtime_value]?

  • @lookahead: runtime_value

  • @lookahead_value: runtime_value

  • @lookahead_location: runtime_value

  • @recovery_shifts: Integer

  • @semantic_error: bool

  • @accept_requested: bool

  • @unknown_token_id: Integer?

  • @unknown_token_name: String?

  • @push_status: :idle | :active | :finished

  • @driver_status: :idle | :pull | :push

  • @runtime_driver_thread: Thread?

  • @runtime_observers: Hash[Observation::Subscription, Proc]?

  • @runtime_event_sequence: Integer

  • @runtime_lookahead_token_display: String?

  • @runtime_observation_mutex: Mutex

  • @repair_policy: RepairPolicy?

  • @repair_input_buffer: Array[RepairInput]?

  • @repair_selected: bool

  • @semantic_locations: Array[runtime_value]?

  • @semantic_location_names: Hash[Symbol, Integer]?

  • @semantic_result_location: runtime_value

  • @trace_value_printer: (^(runtime_value) -> runtime_value)?

  • @sync_recovery_context: Hash[Symbol, runtime_value]?

  • @sync_recovery_token_data: Hash[String, runtime_value]?

  • @sync_recovery_observers: Array[Proc]?

  • @green_builder: CST::GreenBuilder?

  • @green_kinds: CST::Kind?

  • @green_cache: CST::NodeCache?

  • @syntax_root: CST::SyntaxNode?

  • @syntax_diagnostics: Array[runtime_value]

  • @green_pending_skipped: Array[CST::GreenTrivia]

  • @resource_limits: ResourceLimits

  • @recovery_attempts: Integer

  • @runtime_parser_tables: Hash[Symbol, runtime_value]?

  • @runtime_fast_path: bool

  • @runtime_fast_path_tracker_installed: bool

  • @runtime_fast_path_hooks_mutated: bool

  • @runtime_fast_path_singleton_ancestors: Array[Module]?

  • @syntax_only: bool

  • @green_cache_override: CST::NodeCache?

  • @green_token_states: Array[Symbol]

  • @green_memo_stack: Array[CST::ParseMemo::Entry]

  • @green_parse_memo_valid: bool

  • @green_initial_state: Integer

  • @syntax_parse_memo: CST::ParseMemo?

  • @green_reused_right_edge: bool

  • @incremental_reused_descendants: Integer



5996
5997
5998
 
# File 'lib/json5/generated_parser.rb', line 5996

def syntax_parse_memo
  @syntax_parse_memo
end

Class Method Details

.incremental_session(source_text, resource_limits: nil, blender: true) ⇒ Object

Start a syntax-only incremental session backed by a generated lexer. Parser production actions are suppressed, but generated lexer actions still execute. The generated class is trusted application code, not a sandbox.

RBS:

  • (CST::SourceText source_text, ?resource_limits: ResourceLimits?, ?blender: bool) -> CST::IncrementalParseSession



6036
6037
6038
6039
6040
 
# File 'lib/json5/generated_parser.rb', line 6036

def self.incremental_session(source_text, resource_limits: nil, blender: true)
  CST::IncrementalParseSession.new(
    self, source_text, resource_limits: resource_limits, blender: blender
  )
end

.syntax_execution_profile ⇒ Object

Report the trust boundary of syntax-only operations for this loaded parser class. Current generated artifacts can contain user sections and Ruby lexer actions, so they are always trusted application code.



6003
6004
6005
 
# File 'lib/json5/generated_parser.rb', line 6003

def self.syntax_execution_profile
  :trusted_application_code
end

.syntax_session(source, execution_profile: nil, resource_limits: nil, limits: nil, cancellation: nil, blender: true) ⇒ Object

Open a generated-language syntax session backed by the existing incremental Red/Green CST engine. The execution profile must be passed explicitly because generated lexer actions execute in this path.



6011
6012
6013
6014
6015
6016
6017
6018
6019
6020
6021
6022
6023
6024
6025
6026
6027
6028
 
# File 'lib/json5/generated_parser.rb', line 6011

def self.syntax_session(
  source,
  execution_profile: nil,
  resource_limits: nil,
  limits: nil,
  cancellation: nil,
  blender: true
)
  SyntaxSession.new(
    self,
    source,
    execution_profile: execution_profile,
    resource_limits: resource_limits,
    limits: limits,
    cancellation: cancellation,
    blender: blender
  )
end

Instance Method Details

#do_parse ⇒ Object

Pull tokens from next_token and execute parser production actions. A generated-lexer next_token executes lexer actions; a handwritten implementation does not invoke the generated lexer.

RBS:

  • () -> runtime_value



6118
6119
6120
 
# File 'lib/json5/generated_parser.rb', line 6118

def do_parse
  drive_parser(nil)
end

#expected_tokens ⇒ Object

Return token names accepted in the current parser state.

RBS:

  • () -> Array[String]



6337
6338
6339
6340
6341
6342
6343
6344
6345
6346
6347
 
# File 'lib/json5/generated_parser.rb', line 6337

def expected_tokens
  ensure_runtime_initialized!
  return expected_tokens_exact if parser_tables[:exact_expected_tokens]
  return [] if @state_stack.empty?

  state = @state_stack.last
  parser_tables.fetch(:token_names).keys.filter_map do |token_id|
    action = table_lookup(parser_tables.fetch(:actions), state, token_id) || default_action(state) || ERROR_ACTION
    token_to_str(token_id) unless error_action?(action) || token_id == ERROR_TOKEN
  end
end

#expected_tokens_exact ⇒ Object

Return token names that survive all required default reductions. Semantic actions are not evaluated during this lookahead correction.

RBS:

  • () -> Array[String]



6352
6353
6354
6355
6356
6357
6358
6359
6360
6361
 
# File 'lib/json5/generated_parser.rb', line 6352

def expected_tokens_exact
  ensure_runtime_initialized!
  return [] if @state_stack.empty?

  parser_tables.fetch(:token_names).keys.filter_map do |token_id|
    next if token_id == ERROR_TOKEN

    token_to_str(token_id) if exact_lookahead_accepted?(token_id)
  end
end

#finish(location: nil) ⇒ Object

Supply EOF to a caller-driven parser session and return its result. Committed reductions execute parser production actions; this token-fed path does not invoke generated lexer actions.

RBS:

  • (?location: runtime_value) -> runtime_value



6177
6178
6179
6180
6181
6182
6183
6184
6185
6186
6187
6188
6189
6190
6191
6192
6193
6194
6195
6196
6197
 
# File 'lib/json5/generated_parser.rb', line 6177

def finish(location: nil)
  run_push_driver do
    start_push_session
    refresh_runtime_fast_path_after_user_code!
    if @repair_policy
      enqueue_or_assign_repair_input(
        RepairInput.new(token_id: EOF_TOKEN, token_name: token_to_str(EOF_TOKEN), value: nil, location: location)
      )
    else
      @lookahead = EOF_TOKEN
      @lookahead_value = nil
      @lookahead_location = location
      @runtime_fast_path = false unless nil.equal?(location)
      materialize_compatible_lookahead
    end
    outcome = run_push_lookahead
    return outcome.fetch(1) if outcome.is_a?(Array)

    raise ParseError, "(input):1:1: parser requested input after EOF"
  end
end

#loc(reference) ⇒ Object

Return the location of a one-based RHS position or named reference while a semantic action is running.

RBS:

  • (Integer | Symbol | String reference) -> runtime_value

Raises:



6366
6367
6368
6369
6370
6371
6372
6373
6374
6375
6376
6377
6378
6379
6380
6381
 
# File 'lib/json5/generated_parser.rb', line 6366

def loc(reference)
  locations = @semantic_locations
  raise ParseError, "(runtime):1:1: loc is only available inside a semantic action" unless locations

  index = if reference.is_a?(Integer)
            raise ArgumentError, "location index must be positive" unless reference.positive?

            reference - 1
          else
            names = @semantic_location_names || EMPTY_LOCATION_NAMES
            names.fetch(reference.to_sym) do
              raise ArgumentError, "unknown named location #{reference.inspect}"
            end
          end
  locations.fetch(index) { raise ArgumentError, "location index #{reference.inspect} is outside the RHS" }
end

#next_token ⇒ Object

Override in pull parsers. Return [token, value], [token, value, location], false, or nil.

RBS:

  • () -> ([runtime_value, runtime_value] | [runtime_value, runtime_value, runtime_value] | false | nil)

Raises:

  • (NotImplementedError) 


6225
6226
6227
 
# File 'lib/json5/generated_parser.rb', line 6225

def next_token
  raise NotImplementedError, "(input):1:1: next_token must be implemented"
end

#on_discard(_token_id, _value, _location, _reason) ⇒ Object

Called when yacc recovery discards an application token.

RBS:

  • (Integer token_id, runtime_value value, runtime_value location, Symbol reason) -> void



6286
 
# File 'lib/json5/generated_parser.rb', line 6286

def on_discard(_token_id, _value, _location, _reason); end

#on_error(token_id, value, _value_stack) ⇒ Object

Override to recover from syntax errors. The default raises unless a bounded automatic repair has already been selected.

RBS:

  • (Integer token_id, runtime_value value, Array[runtime_value] value_stack) -> runtime_value

Raises:



6232
6233
6234
6235
6236
6237
6238
6239
6240
6241
6242
6243
6244
6245
6246
6247
6248
6249
6250
6251
6252
 
# File 'lib/json5/generated_parser.rb', line 6232

def on_error(token_id, value, _value_stack)
  return if @repair_selected
  return if cst_enabled?

  expected = expected_tokens
  token_name = token_to_str(token_id)
  state = @state_stack.last
  configured = parser_tables.fetch(:error_messages, EMPTY_ROW)[state]
  error_id, detail = configured_error_message(configured)
  raise ParseError.new(
    token_id: token_id,
    token_name: token_name,
    token_value: value,
    expected_tokens: expected,
    location: @lookahead_location,
    state: state,
    suggestions: token_suggestions(token_name, expected),
    error_id: error_id,
    detail: detail
  )
end

#on_error_recover(_token_id, _value, _value_stack) ⇒ Object

Called after the synthetic error token enters a recovery state. The payload describes the original error before recovery popped stacks.

RBS:

  • (Integer token_id, runtime_value value, Array[runtime_value] value_stack) -> void



6277
 
# File 'lib/json5/generated_parser.rb', line 6277

def on_error_recover(_token_id, _value, _value_stack); end

#on_error_recover_location(_token_id, _value, _value_stack, _location, _state) ⇒ Object

Location-aware recovery observer.

RBS:

  • (Integer token_id, runtime_value value, Array[runtime_value] value_stack, runtime_value location, Integer state) -> void



6282
 
# File 'lib/json5/generated_parser.rb', line 6282

def on_error_recover_location(_token_id, _value, _value_stack, _location, _state); end

#on_reduce(_production_id, _values, _result) ⇒ Object Also known as: __ibex_fast_path_on_reduce

Called after a production's semantic action and goto are committed. Override to observe its id, RHS values, and semantic result.

RBS:

  • (Integer production_id, Array[runtime_value] values, runtime_value result) -> void



6267
 
# File 'lib/json5/generated_parser.rb', line 6267

def on_reduce(_production_id, _values, _result); end

#on_reduce_location(_production_id, _values, _result, _locations, _result_location) ⇒ Object Also known as: __ibex_fast_path_on_reduce_location

Location-aware reduction observer.

RBS:

  • (Integer production_id, Array[runtime_value] values, runtime_value result, Array[runtime_value] locations, runtime_value result_location) -> void



6272
 
# File 'lib/json5/generated_parser.rb', line 6272

def on_reduce_location(_production_id, _values, _result, _locations, _result_location); end

#on_repair(_plan) ⇒ Object

Called once after a repair is selected and before its edited token prefix is replayed through normal parser actions.

RBS:

  • (RepairPlan plan) -> void



6302
 
# File 'lib/json5/generated_parser.rb', line 6302

def on_repair(_plan); end

#on_shift(_token_id, _value, _state) ⇒ Object Also known as: __ibex_fast_path_on_shift

Called after an ordinary input token is shifted. Override to observe the internal token id, semantic value, and destination state.

RBS:

  • (Integer token_id, runtime_value value, Integer state) -> void



6257
 
# File 'lib/json5/generated_parser.rb', line 6257

def on_shift(_token_id, _value, _state); end

#on_shift_location(_token_id, _value, _state, _location) ⇒ Object Also known as: __ibex_fast_path_on_shift_location

Location-aware shift observer. The compatible hook above retains its original signature and runs first.

RBS:

  • (Integer token_id, runtime_value value, Integer state, runtime_value location) -> void



6262
 
# File 'lib/json5/generated_parser.rb', line 6262

def on_shift_location(_token_id, _value, _state, _location); end

#parse_with_syntax ⇒ Object

Parse through next_token and return both the semantic value and Red root. Parser production actions execute. Generated lexer actions execute only when next_token is supplied by the generated lexer.

RBS:

  • () -> CST::ParseResult



6126
6127
6128
 
# File 'lib/json5/generated_parser.rb', line 6126

def parse_with_syntax
  syntax_parse_result(do_parse)
end

#push(token, value = nil, location = nil) ⇒ Object

Supply one token to a caller-driven parser session. Committed reductions execute parser production actions; this token-fed path does not invoke generated lexer actions. Returns :need_more after consuming it, [:accepted, result] after acceptance, or [:rejected, result] after recovery terminates.

RBS:

  • (runtime_value token, ?runtime_value value, ?runtime_value location) -> runtime_value

Raises:



6154
6155
6156
6157
6158
6159
6160
6161
6162
6163
6164
6165
6166
6167
6168
6169
6170
6171
 
# File 'lib/json5/generated_parser.rb', line 6154

def push(token, value = nil, location = nil)
  raise ParseError, "(input):1:1: push requires a token; call finish for EOF" if token.nil? || token == false

  run_push_driver do
    start_push_session
    refresh_runtime_fast_path_after_user_code!
    if @repair_policy
      enqueue_or_assign_repair_input(repair_input(token, value, location))
    else
      @lookahead = internal_token_id(token)
      @lookahead_value = value
      @lookahead_location = location
      @runtime_fast_path = false unless nil.equal?(location)
      materialize_compatible_lookahead
    end
    run_push_lookahead
  end
end

#repair_policy ⇒ Object

RBS:

  • () -> RepairPolicy?



6068
6069
6070
6071
 
# File 'lib/json5/generated_parser.rb', line 6068

def repair_policy
  ensure_runtime_initialized!
  @repair_policy
end

#repair_policy=(policy) ⇒ Object

Enable bounded automatic repair for the next parser session. Assign nil to restore the compatible yacc-only behavior.

RBS:

  • (RepairPolicy? policy) -> RepairPolicy?



6082
6083
6084
6085
6086
6087
6088
6089
6090
6091
6092
6093
6094
6095
6096
 
# File 'lib/json5/generated_parser.rb', line 6082

def repair_policy=(policy)
  ensure_runtime_initialized!
  unless policy.nil? || policy.is_a?(RepairPolicy)
    raise ArgumentError, "repair_policy must be an Ibex::Runtime::RepairPolicy or nil"
  end

  @runtime_observation_mutex.synchronize do
    ensure_driver_available_without_lock!
    if @push_status == :active
      raise ParseError, "(repair):1:1: repair_policy cannot change during an active push session"
    end

    @repair_policy = policy
  end
end

#reset_push ⇒ Object

Discard a caller-driven session so this parser can accept a new one.

RBS:

  • () -> nil



6201
6202
6203
6204
6205
6206
6207
6208
6209
6210
6211
6212
6213
6214
6215
6216
6217
6218
6219
6220
 
# File 'lib/json5/generated_parser.rb', line 6201

def reset_push
  ensure_runtime_initialized!
  @runtime_observation_mutex.synchronize do
    ensure_driver_available_without_lock!
    @push_status = :idle
    @source = nil
    @state_stack = []
    install_value_stack([])
    @location_stack = nil
    @lookahead = NO_LOOKAHEAD
    @lookahead_value = nil
    @lookahead_location = nil
    @runtime_lookahead_token_display = nil
    @repair_input_buffer = nil
    @repair_selected = false
    @runtime_parser_tables = nil
    @runtime_fast_path = false
  end
  nil
end

#resource_limits ⇒ Object

RBS:

  • () -> ResourceLimits



6074
6075
6076
6077
 
# File 'lib/json5/generated_parser.rb', line 6074

def resource_limits
  ensure_runtime_initialized!
  @resource_limits
end

#resource_limits=(limits) ⇒ Object

Replace the limits used by future sessions.

RBS:

  • (ResourceLimits limits) -> ResourceLimits



6100
6101
6102
6103
6104
6105
6106
6107
6108
6109
6110
6111
6112
 
# File 'lib/json5/generated_parser.rb', line 6100

def resource_limits=(limits)
  ensure_runtime_initialized!
  validate_resource_limits!(limits)

  @runtime_observation_mutex.synchronize do
    ensure_driver_available_without_lock!
    if @push_status == :active
      raise ParseError, "(resource):1:1: resource_limits cannot change during an active push session"
    end

    @resource_limits = limits
  end
end

#result_loc ⇒ Object

Return the synthesized span of the reduction being evaluated.

RBS:

  • () -> runtime_value



6385
6386
6387
6388
6389
6390
6391
 
# File 'lib/json5/generated_parser.rb', line 6385

def result_loc
  unless @semantic_locations
    raise ParseError, "(runtime):1:1: result_loc is only available inside a semantic action"
  end

  @semantic_result_location
end

#syntax_root ⇒ Object

Return the Red source-file root built by the most recent CST parse.

RBS:

  • () -> CST::SyntaxNode?



6132
6133
6134
6135
 
# File 'lib/json5/generated_parser.rb', line 6132

def syntax_root
  ensure_runtime_initialized!
  @syntax_root
end

#token_to_str(token_id) ⇒ Object Also known as: __ibex_fast_path_token_to_str

Return a human-readable name for an internal token id.

RBS:

  • (Integer token_id) -> String



6306
6307
6308
6309
6310
 
# File 'lib/json5/generated_parser.rb', line 6306

def token_to_str(token_id)
  return @unknown_token_name || token_id.to_s if token_id == @unknown_token_id

  parser_tables.fetch(:token_names).fetch(token_id, token_id.to_s)
end

#trace_value_printer=(printer) ⇒ Object

Install an opt-in value formatter for human-readable yydebug traces.

RBS:

  • ((^(runtime_value) -> runtime_value)? printer) -> (^(runtime_value) -> runtime_value)?



6290
6291
6292
6293
6294
6295
6296
6297
 
# File 'lib/json5/generated_parser.rb', line 6290

def trace_value_printer=(printer)
  ensure_runtime_initialized!
  unless printer.nil? || printer.respond_to?(:call)
    raise ArgumentError, "trace value printer must respond to call or be nil"
  end

  @trace_value_printer = printer
end

#yyaccept ⇒ Object

Accept immediately after the current semantic action completes.

RBS:

  • () -> nil



6329
6330
6331
6332
6333
 
# File 'lib/json5/generated_parser.rb', line 6329

def yyaccept
  @runtime_fast_path = false
  @accept_requested = true
  nil
end

#yydebug ⇒ Object

RBS:

  • () -> bool



6049
6050
6051
6052
 
# File 'lib/json5/generated_parser.rb', line 6049

def yydebug
  ensure_runtime_initialized!
  @yydebug
end

#yydebug=(enabled) ⇒ Object

RBS:

  • (bool enabled) -> bool



6055
6056
6057
6058
6059
 
# File 'lib/json5/generated_parser.rb', line 6055

def yydebug=(enabled)
  ensure_runtime_initialized!
  @runtime_fast_path = false
  @yydebug = enabled
end

#yydebug_output=(output) ⇒ Object

RBS:

  • (IO output) -> IO



6062
6063
6064
6065
 
# File 'lib/json5/generated_parser.rb', line 6062

def yydebug_output=(output)
  ensure_runtime_initialized!
  @yydebug_output = output
end

#yyerrok ⇒ Object

Leave error recovery immediately.

RBS:

  • () -> nil



6322
6323
6324
6325
 
# File 'lib/json5/generated_parser.rb', line 6322

def yyerrok
  @recovery_shifts = 0
  nil
end

#yyerror ⇒ Object

Enter error recovery from a semantic action without calling on_error.

RBS:

  • () -> nil



6314
6315
6316
6317
6318
 
# File 'lib/json5/generated_parser.rb', line 6314

def yyerror
  @runtime_fast_path = false
  @semantic_error = true
  nil
end

#yyparse(receiver, method_id) ⇒ Object

Parse tokens yielded by receiver.method_id and execute parser production actions. This caller-fed path does not invoke generated lexer actions.

RBS:

  • (runtime_value receiver, Symbol method_id) -> runtime_value



6141
6142
6143
6144
6145
6146
 
# File 'lib/json5/generated_parser.rb', line 6141

def yyparse(receiver, method_id)
  stream = Enumerator.new do |tokens|
    receiver.__send__(method_id) { |token| tokens << token }
  end
  drive_parser(-> { stream.next })
end