Class: Kotoshu::Languages::Japanese::POSTagger

Inherits:
Components::PosTagger show all
Defined in:
lib/kotoshu/languages/ja/language.rb

Overview

Japanese POS tagger using morphological analysis.

Japanese POS tagging is integrated with tokenization via Suika gem, which provides both segmentation and part-of-speech information.

Suika output format: surfacePOS,subcat1,subcat2,subcat3,conj_type,conj_form,lemma,reading,pronunciation Example: "すもも\t名詞,一般,,,,,すもも,スモモ,スモモ"

POS tags use universal English categories for common types, and ROMAJI (Latin script) identifiers based on Japanese terminology only for language-specific categories without universal equivalents.

Constant Summary collapse

FLAG_TO_POS =

Japanese POS tag mappings from Suika to standard identifiers.

Strategy: Use universal English POS tags (NOUN, VERB, etc.) with English suffixes for subcategories. All identifiers are ASCII.

Main categories (field 0) - universal:

  • 名詞 → NOUN
  • 動詞 → VERB
  • 助詞 → PARTICLE
  • 助動詞 → AUX

Noun subcategories (field 1):

  • NOUN_COMMON: 一般 - common nouns
  • NOUN_PROPER: 固有名詞 - proper nouns
  • NOUN_PROPER_GEOGRAPHIC: 固有名詞,地域 - proper noun, geographic
  • NOUN_SUFFIX: 接尾 - suffixes
  • NOUN_DEPENDENT: 非自立 - dependent nouns (cannot stand alone)
  • NOUN_SA_CONNECTION: サ変接続 - sa-variant connection nouns

Particle subcategories (field 1):

  • PARTICLE_GRAMMAR: 格助詞 - grammar/case particles (が, を, に, etc.)
  • PARTICLE_BINDING: 係助詞 - binding particles (は, も, etc.)
  • PARTICLE_ADNOMINAL: 連体化 - adnominal particles (の)

Verb subcategories (field 1):

  • VERB_INDEPENDENT: 自立 - independent verbs
{
  # Main categories - universal English
  '名詞' => 'NOUN',
  '動詞' => 'VERB',
  '助詞' => 'PARTICLE',
  '助動詞' => 'AUX',

  # Noun subcategories
  '名詞,一般' => 'NOUN_COMMON',
  '名詞,固有名詞' => 'NOUN_PROPER',
  '名詞,固有名詞,地域' => 'NOUN_PROPER_GEOGRAPHIC',
  '名詞,接尾' => 'NOUN_SUFFIX',
  '名詞,非自立' => 'NOUN_DEPENDENT',
  '名詞,サ変接続' => 'NOUN_SA_CONNECTION',

  # Particle subcategories
  '助詞,格助詞' => 'PARTICLE_GRAMMAR',
  '助詞,係助詞' => 'PARTICLE_BINDING',
  '助詞,連体化' => 'PARTICLE_ADNOMINAL',

  # Verb subcategories
  '動詞,自立' => 'VERB_INDEPENDENT',
}.freeze

Instance Method Summary collapse

Methods inherited from Components::PosTagger

#tag_word

Constructor Details

#initialize(dictionary_path: nil, flag_mapping: FLAG_TO_POS) ⇒ POSTagger

Returns a new instance of POSTagger.



222
223
224
225
226
227
 
# File 'lib/kotoshu/languages/ja/language.rb', line 222

def initialize(dictionary_path: nil, flag_mapping: FLAG_TO_POS)
  @dictionary_path = dictionary_path
  @flag_mapping = flag_mapping
  @suika_tagger = nil
  @lookup_cache = {}
end

Instance Method Details

#clear_cache ⇒ Object 



255
256
257
 
# File 'lib/kotoshu/languages/ja/language.rb', line 255

def clear_cache
  @lookup_cache.clear
end

#flag_mapping ⇒ Object 



247
248
249
 
# File 'lib/kotoshu/languages/ja/language.rb', line 247

def flag_mapping
  @flag_mapping
end

#flag_mapping=(mapping) ⇒ Object 



251
252
253
 
# File 'lib/kotoshu/languages/ja/language.rb', line 251

def flag_mapping=(mapping)
  @flag_mapping = mapping
end

#tag(tokens) ⇒ Object 



229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
 
# File 'lib/kotoshu/languages/ja/language.rb', line 229

def tag(tokens)
  return [] if tokens.nil? || tokens.empty?

  # Suika::Tagger is process-wide memoized in Language::Suika;
  # raises SuikaUnavailable when the gem is missing.
  @suika_tagger ||= Kotoshu::Language::Suika.tagger

  tokens.map do |token|
    word = token[:token]
    if word.nil? || word.empty?
      token.merge(pos_tag: nil, lemma: nil)
    else
      lookup_result = lookup_with_pos(word)
      token.merge(pos_tag: lookup_result[:pos_tag], lemma: lookup_result[:lemma] || word)
    end
  end
end