Module: JLPT::CorpusStats

Defined in:
lib/jlpt/analyzers/corpus_stats.rb

Overview

Corpus Lexical Diversity and Statistical Analyzer.

Computes Type-Token Ratio (TTR), total word count, unique word count, and hapax legomena count for text.

Class Method Summary collapse

Class Method Details

.analyze(text) ⇒ Hash

Analyze text corpus stats and lexical diversity (TTR)

Parameters:

  • text (String) —

    Japanese text

Returns:

  • (Hash) —

    corpus statistics hash



15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 
# File 'lib/jlpt/analyzers/corpus_stats.rb', line 15

def analyze(text)
  cleaned = Preprocessor.clean(text)
  return empty_stats if cleaned.empty?

  tokens = Tokenizer.lemmata(cleaned)
  return empty_stats if tokens.empty?

  total = tokens.length
  counts = tokens.tally
  unique = counts.keys.length
  hapax = counts.values.count(1)
  ttr = (unique.to_f / total).round(2)

  { total_words: total, unique_words: unique, hapax_count: hapax, ttr: ttr }
end