Module: JLPT::VocabClassifier

Defined in:
lib/jlpt/core/vocab_classifier.rb

Overview

Vocabulary Classifier for JLPT Levels.

Analyzes tokens in Japanese text and categorizes vocabulary into JLPT levels (N5 to N1) or Non-JLPT, computing distribution statistics and percentages.

Class Method Summary collapse

Class Method Details

.classify(text) ⇒ Hash{Symbol => Array<String>}

Classify all vocabulary tokens in text by JLPT level

Parameters:

  • text (String) —

    Japanese text

Returns:

  • (Hash{Symbol => Array<String>}) —

    level to vocabulary lemmata mapping



15
16
17
18
19
20
21
22
 
# File 'lib/jlpt/core/vocab_classifier.rb', line 15

def classify(text)
  result = { n5: [], n4: [], n3: [], n2: [], n1: [], non_jlpt: [] }
  Tokenizer.lemmata(text).each do |lemma|
    lvl = Dictionary.vocab_level(lemma)
    result.key?(lvl) ? result[lvl] << lemma : result[:non_jlpt] << lemma
  end
  result
end

.distribution(text) ⇒ Hash{Symbol => Float}

Calculate percentage distribution of vocabulary across JLPT levels

Parameters:

  • text (String) —

    Japanese text

Returns:

  • (Hash{Symbol => Float}) —

    level to percentage mapping



28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/jlpt/core/vocab_classifier.rb', line 28

def distribution(text)
  classified = classify(text)
  total_words = classified.values.sum(&:length)
  return empty_distribution if total_words.zero?

  dist = {}
  classified.each do |lvl, words|
    pct = (words.length.to_f / total_words * 100).round(2)
    dist[lvl] = pct
  end
  dist
end