Class: Ibex::Runtime::CST::TokenMemo

Inherits:
Object
  • Object
show all
Defined in:
lib/ibex/runtime/cst/incremental/token_memo.rb,
sig/ibex/runtime/cst/incremental/token_memo.rbs

Overview

Session-local token metadata used to validate lexical resynchronization.

Constant Summary collapse

ENTRY_BYTES =

Signature:

  • Integer

Returns:

  • (Integer) 
32

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(tokens:, offsets:, states:) ⇒ TokenMemo

Returns a new instance of TokenMemo.

RBS:

  • (tokens: Array[GreenToken], offsets: Array[Integer], states: Array[Symbol]) -> void

Parameters:

  • tokens: (Array[GreenToken])
  • offsets: (Array[Integer])
  • states: (Array[Symbol]) 


16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 16

def initialize(tokens:, offsets:, states:)
  unless tokens.length == offsets.length && offsets.length == states.length
    raise ArgumentError, "token memo arrays must have equal lengths"
  end

  ordered = (0...[offsets.length - 1, 0].max).all? do |index|
    offsets.fetch(index) <= offsets.fetch(index + 1)
  end
  raise ArgumentError, "token memo offsets must be ordered" unless ordered

  @tokens = tokens.dup.freeze
  @offsets = offsets.dup.freeze
  @states = states.dup.freeze
  freeze
end

Instance Attribute Details

#offsets ⇒ Array[Integer] (readonly)

Signature:

  • Array[Integer]

Returns:

  • (Array[Integer]) 


12
13
14
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 12

def offsets
  @offsets
end

#states ⇒ Array[Symbol] (readonly)

Signature:

  • Array[Symbol]

Returns:

  • (Array[Symbol]) 


13
14
15
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 13

def states
  @states
end

#tokens ⇒ Array[GreenToken] (readonly)

Signature:

  • Array[GreenToken]

Returns:



11
12
13
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 11

def tokens
  @tokens
end

Class Method Details

.from_root(root, states: []) ⇒ TokenMemo

RBS:

  • (SyntaxNode root, ?states: Array[Symbol]) -> TokenMemo

Parameters:

  • root (SyntaxNode)
  • states: (Array[Symbol]) (defaults to: [])

Returns:



33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 33

def self.from_root(root, states: [])
  tokens = root.tokens.map(&:green)
  offsets = [] #: Array[Integer]
  offset = 0
  tokens.each do |token|
    offsets << offset
    offset += token.full_width
  end
  actual_states = Array.new(tokens.length) { |index| states.fetch(index, :INITIAL) }
  new(tokens: tokens, offsets: offsets, states: actual_states)
end

Instance Method Details

#damage_index(edits) ⇒ Integer?

The first token whose owned full span contains or follows an edit.

RBS:

  • (Array[TextEdit] edits) -> Integer?

Parameters:

Returns:

  • (Integer, nil) 


47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 47

def damage_index(edits)
  normalized = TextEdit.normalize(edits)
  edit = normalized.first
  return unless edit

  point = edit.start
  @tokens.each_index do |index|
    start = @offsets.fetch(index)
    finish = start + @tokens.fetch(index).full_width
    return index if point < finish || (start == point && finish == point)
  end
  @tokens.empty? ? nil : @tokens.length - 1
end

#estimated_bytes ⇒ Integer

RBS:

  • () -> Integer

Returns:

  • (Integer) 


62
 
# File 'lib/ibex/runtime/cst/incremental/token_memo.rb', line 62

def estimated_bytes = @tokens.length * ENTRY_BYTES