Module: JLPT::Engine::PureRubyFallback

Defined in:
lib/jlpt/core/engine.rb

Overview

Pure Ruby Fallback Engine for environments without MeCab installed

Constant Summary collapse

KANJI_KATAKANA_REGEX = 
/[\u4E00-\u9FFF\u30A0-\u30FF]/.freeze
HIRAGANA_REGEX = 
/[\u3040-\u309F]/.freeze
WORD_SCAN_REGEX = 
/[\u4E00-\u9FFF]+|[\u3040-\u309F]+|[\u30A0-\u30FF]+|[a-zA-Z0-9]+|\S/.freeze

Class Method Summary collapse

Class Method Details

.parse(text) ⇒ Object 



84
85
86
87
88
 
# File 'lib/jlpt/core/engine.rb', line 84

def parse(text)
  return [] if text.nil? || text.to_s.strip.empty?

  text.scan(WORD_SCAN_REGEX).flat_map { |token| process_fallback_token(token) }
end