Module: Varar::Core::Offsets

Defined in:
lib/varar/core/span.rb

Overview

UTF-16 offset conversion. Ruby strings are code-point indexed, so the whole core converts to/from UTF-16 code units here (the single riskiest part of the port — mirrors Python's span.py). Reused by the matcher and by hash.rb.

Class Method Summary collapse

Class Method Details

.cp_index_for_utf16(source, u16) ⇒ Object

Inverse of to_utf16_offset: the code-point index at a UTF-16 offset.



34
35
36
37
38
39
40
41
42
 
# File 'lib/varar/core/span.rb', line 34

def cp_index_for_utf16(source, u16)
  count = 0
  source.each_char.with_index do |ch, i|
    return i if count >= u16

    count += ch.ord > 0xFFFF ? 2 : 1
  end
  source.length
end

.line_col(source, offset_u16) ⇒ Object

1-based [line, col] at a UTF-16 offset; col counts UTF-16 units and resets to 1 after each newline (mirrors span.ts's lineCol).



53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/varar/core/span.rb', line 53

def line_col(source, offset_u16)
  line = 1
  col = 1
  count = 0
  source.each_char do |ch|
    break if count >= offset_u16

    width = ch.ord > 0xFFFF ? 2 : 1
    if ch == "\n"
      line += 1
      col = 1
    else
      col += width
    end
    count += width
  end
  [line, col]
end

.span_from_offsets(source, start_u16, end_u16) ⇒ Object

Build a Span from UTF-16 start/end offsets.



73
74
75
76
77
78
79
80
81
 
# File 'lib/varar/core/span.rb', line 73

def span_from_offsets(source, start_u16, end_u16)
  start_line, start_col = line_col(source, start_u16)
  end_line, end_col = line_col(source, end_u16)
  Span.new(
    start_offset: start_u16, end_offset: end_u16,
    start_line: start_line, start_col: start_col,
    end_line: end_line, end_col: end_col
  )
end

.to_utf16_offset(source, cp_index) ⇒ Object

UTF-16 offset of the code-point index cp_index in source.



29
30
31
 
# File 'lib/varar/core/span.rb', line 29

def to_utf16_offset(source, cp_index)
  utf16_len(source[0...cp_index])
end

.utf16_len(str) ⇒ Object

UTF-16 code-unit length of a string (astral chars count as 2).



22
23
24
25
26
 
# File 'lib/varar/core/span.rb', line 22

def utf16_len(str)
  n = 0
  str.each_char { |ch| n += ch.ord > 0xFFFF ? 2 : 1 }
  n
end

.utf16_slice(source, start_u16, end_u16) ⇒ Object

Slice source by UTF-16 offsets, returning the covered substring.



45
46
47
48
49
 
# File 'lib/varar/core/span.rb', line 45

def utf16_slice(source, start_u16, end_u16)
  a = cp_index_for_utf16(source, start_u16)
  b = cp_index_for_utf16(source, end_u16)
  source[a...b]
end