Class: Ibex::Runtime::CST::TokenMemo

Inherits:
Object
  • Object
show all
Defined in:
lib/json5/generated_parser.rb

Overview

Session-local token metadata used to validate lexical resynchronization.

Constant Summary collapse

ENTRY_BYTES =

Signature:

  • Integer

32

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(tokens:, offsets:, states:) ⇒ TokenMemo

Returns a new instance of TokenMemo.

RBS:

  • (tokens: Array[GreenToken], offsets: Array[Integer], states: Array[Symbol]) -> void

Raises:

  • (ArgumentError) 


2735
2736
2737
2738
2739
2740
2741
2742
2743
2744
2745
2746
2747
2748
2749
 
# File 'lib/json5/generated_parser.rb', line 2735

def initialize(tokens:, offsets:, states:)
  unless tokens.length == offsets.length && offsets.length == states.length
    raise ArgumentError, "token memo arrays must have equal lengths"
  end

  ordered = (0...[offsets.length - 1, 0].max).all? do |index|
    offsets.fetch(index) <= offsets.fetch(index + 1)
  end
  raise ArgumentError, "token memo offsets must be ordered" unless ordered

  @tokens = tokens.dup.freeze
  @offsets = offsets.dup.freeze
  @states = states.dup.freeze
  freeze
end

Instance Attribute Details

#offsets ⇒ Object (readonly)

Signature:

  • Array[Integer]



2731
2732
2733
 
# File 'lib/json5/generated_parser.rb', line 2731

def offsets
  @offsets
end

#states ⇒ Object (readonly)

Signature:

  • Array[Symbol]



2732
2733
2734
 
# File 'lib/json5/generated_parser.rb', line 2732

def states
  @states
end

#tokens ⇒ Object (readonly)

Signature:

  • Array[GreenToken]



2730
2731
2732
 
# File 'lib/json5/generated_parser.rb', line 2730

def tokens
  @tokens
end

Class Method Details

.from_root(root, states: []) ⇒ Object

RBS:

  • (SyntaxNode root, ?states: Array[Symbol]) -> TokenMemo



2752
2753
2754
2755
2756
2757
2758
2759
2760
2761
2762
 
# File 'lib/json5/generated_parser.rb', line 2752

def self.from_root(root, states: [])
  tokens = root.tokens.map(&:green)
  offsets = [] #: Array[Integer]
  offset = 0
  tokens.each do |token|
    offsets << offset
    offset += token.full_width
  end
  actual_states = Array.new(tokens.length) { |index| states.fetch(index, :INITIAL) }
  new(tokens: tokens, offsets: offsets, states: actual_states)
end

Instance Method Details

#damage_index(edits) ⇒ Object

The first token whose owned full span contains or follows an edit.

RBS:

  • (Array[TextEdit] edits) -> Integer?



2766
2767
2768
2769
2770
2771
2772
2773
2774
2775
2776
2777
2778
 
# File 'lib/json5/generated_parser.rb', line 2766

def damage_index(edits)
  normalized = TextEdit.normalize(edits)
  edit = normalized.first
  return unless edit

  point = edit.start
  @tokens.each_index do |index|
    start = @offsets.fetch(index)
    finish = start + @tokens.fetch(index).full_width
    return index if point < finish || (start == point && finish == point)
  end
  @tokens.empty? ? nil : @tokens.length - 1
end

#estimated_bytes ⇒ Object

RBS:

  • () -> Integer



2781
 
# File 'lib/json5/generated_parser.rb', line 2781

def estimated_bytes = @tokens.length * ENTRY_BYTES