Module: JLPT::PosProfiler

Defined in:
lib/jlpt/analyzers/pos_profiler.rb

Overview

Part-of-Speech (POS) Distribution Profiler.

Analyzes Japanese tokens and calculates percentage breakdown per POS category (nouns, verbs, adjectives, particles, adverbs, symbols).

Constant Summary collapse

POS_LABELS = 
{
  '名詞' => :noun,
  '動詞' => :verb,
  '形容詞' => :adjective,
  '助詞' => :particle,
  '副詞' => :adverb,
  '記号' => :symbol
}.freeze

Class Method Summary collapse

Class Method Details

.profile(text) ⇒ Hash

Analyze text POS distribution

Parameters:

  • text (String) —

    Japanese text

Returns:

  • (Hash) —

    POS breakdown hash with counts and distribution



24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/jlpt/analyzers/pos_profiler.rb', line 24

def profile(text)
  tokens = Tokenizer.tokenize(text)
  return empty_profile if tokens.empty?

  counts = Hash.new(0)
  tokens.each do |token|
    category = POS_LABELS[token[:pos]] || :other
    counts[category] += 1
  end

  total = tokens.length
  dist = counts.transform_values { |cnt| (cnt.to_f / total * 100).round(2) }

  { counts: counts, distribution: dist, total_tokens: total }
end