Class: SasLexer::Lexer

Inherits:
Object
  • Object
show all
Extended by:
FFI::Library
Defined in:
lib/sas_lexer/lexer.rb

Overview

Ruby wrapper around the ‘sas-lexer` Rust crate, accessed through the small C ABI shim built from `ffi-wrapper/`.

Loads the prebuilt shared library shipped under ‘lib/native/` (or the dev-build flat path produced by `rake sas_lexer:install`).

Defined Under Namespace

Modules: ErrorCode, TokenChannel, TokenType Classes: Token

Constant Summary collapse

LIBRARY_PATH =

Probe order:

1. `lib/native/<host_cpu>-<host_os>/libsas_lexer_ffi.<ext>` —
   prebuilt artifact shipped inside the published universal gem
   for the host's exact platform.
2. `lib/native/libsas_lexer_ffi.<ext>` — flat path produced
   by `bundle exec rake sas_lexer:install` for local
   development.

[
  File.join(lib_native_dir, host_platform, "libsas_lexer_ffi.#{library_ext}"),
  File.join(lib_native_dir, "libsas_lexer_ffi.#{library_ext}"),
].find { |path| File.exist?(path) }

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Lexer

Returns a new instance of Lexer.

Raises:



407
408
409
410
 
# File 'lib/sas_lexer/lexer.rb', line 407

def initialize
  @lexer_ptr = self.class.sas_lexer_new
  raise SasLexer::Error, "Failed to create SAS lexer" if @lexer_ptr.null?
end

Instance Method Details

#finalize ⇒ Object 



472
473
474
 
# File 'lib/sas_lexer/lexer.rb', line 472

def finalize
  free
end

#free ⇒ Object 



465
466
467
468
469
470
 
# File 'lib/sas_lexer/lexer.rb', line 465

def free
  return if @lexer_ptr.null?

  self.class.sas_lexer_free(@lexer_ptr)
  @lexer_ptr = FFI::Pointer::NULL
end

#tokenize(sas_code) ⇒ Object

Raises:



412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
 
# File 'lib/sas_lexer/lexer.rb', line 412

def tokenize(sas_code)
  raise SasLexer::Error, "Lexer has been freed" if @lexer_ptr.null?
  raise SasLexer::Error, "Null pointer provided" if sas_code.nil?

  result = self.class.sas_lexer_tokenize(@lexer_ptr, sas_code)

  if result != ErrorCode::SUCCESS
    error_msg = get_last_error_message
    raise SasLexer::Error, error_msg || "Failed to tokenize SAS code (error code: #{result})"
  end

  token_count = self.class.sas_lexer_token_count(@lexer_ptr)

  tokens = []

  (0...token_count).each do |index|
    text_ptr = self.class.sas_lexer_get_token_text(@lexer_ptr, index)

    next if text_ptr.null?

    text = text_ptr.read_string

    token_struct = Token.new
    token_result = self.class.sas_lexer_get_token(@lexer_ptr, index, token_struct)

    if token_result == ErrorCode::SUCCESS
      tokens << {
        index: index,
        text: text,
        type: token_struct[:token_type],
        channel: token_struct[:channel],
        start: token_struct[:start],
        end: token_struct[:end],
        start_line: token_struct[:start_line],
        end_line: token_struct[:end_line],
        start_column: token_struct[:start_column],
        end_column: token_struct[:end_column]
      }
    else
      tokens << {
        index: index,
        text: text,
        type: nil,
        channel: nil
      }
    end

    self.class.sas_lexer_free_string(text_ptr)
  end

  tokens
end