Module: JLPT::Tokenizer

Defined in:
lib/jlpt/core/tokenizer.rb

Overview

Tokenizer and Morphological Analyzer.

Processes raw text through Preprocessor and Engine to extract structured tokens with lemmata, Part-Of-Speech (POS) categories, and dictionary forms.

Constant Summary collapse

POS_MAP = 
{
  '名詞' => :noun,
  '動詞' => :verb,
  '形容詞' => :i_adjective,
  '形状詞' => :na_adjective,
  '副詞' => :adverb,
  '助詞' => :particle,
  '助動詞' => :auxiliary_verb,
  '連体詞' => :pre_noun_adjective,
  '感動詞' => :interjection,
  '接続詞' => :conjunction,
  '記号' => :symbol
}.freeze

Class Method Summary collapse

Class Method Details

.lemmata(text, filter_pos: nil) ⇒ Array<String>

Extract dictionary forms (lemmata) of words

Parameters:

  • text (String) —

    Japanese text

  • filter_pos (Array<Symbol>) (defaults to: nil) —

    optional POS filter (e.g. [:noun, :verb])

Returns:

  • (Array<String>) —

    array of dictionary forms



42
43
44
45
46
 
# File 'lib/jlpt/core/tokenizer.rb', line 42

def lemmata(text, filter_pos: nil)
  tokens = tokenize(text)
  tokens = tokens.select { |t| filter_pos.include?(t[:pos_category]) } if filter_pos && !filter_pos.empty?
  tokens.map { |t| t[:dictionary_form] }.compact.reject(&:empty?)
end

.tokenize(text) ⇒ Array<Hash>

Tokenize text into structured token objects

Parameters:

  • text (String) —

    Japanese input text

Returns:

  • (Array<Hash>) —

    array of token hashes



29
30
31
32
33
34
35
 
# File 'lib/jlpt/core/tokenizer.rb', line 29

def tokenize(text)
  cleaned = Preprocessor.clean(text)
  return [] if cleaned.empty?

  raw_nodes = Engine.parse(cleaned)
  raw_nodes.map { |node| format_token(node) }
end