Class: Kotoshu::Language::Normalizer::Base

Inherits:
Object
  • Object
show all
Defined in:
lib/kotoshu/language/normalizer/base.rb

Overview

Abstract base class for text normalizers.

Normalizers transform text to a standard form for comparison. Different languages use different normalization strategies.

Examples of normalization:

  • Accent removal (café -> cafe)
  • Case folding (Hello -> hello)
  • Whitespace normalization
  • Punctuation normalization

Examples:

Implement a normalizer

class MyNormalizer < Normalizer::Base
  def normalize(text)
    super.downcase.gsub(/[áàâä]/, 'a')
  end
end

Direct Known Subclasses

Arabic, Hebrew

Instance Method Summary collapse

Instance Method Details

#normalize(text, options = {}) ⇒ String

Normalize text.

Default implementation:

  • Strip leading/trailing whitespace
  • Collapse multiple whitespace to single space
  • Downcase (optional)

Parameters:

  • text (String) —

    Text to normalize

  • options (Hash) (defaults to: {}) —

    Normalization options

Options Hash (options):

  • :downcase (Boolean) — default: true —

    Convert to lowercase

  • :strip_punct (Boolean) — default: false —

    Remove punctuation

  • :collapse_ws (Boolean) — default: true —

    Collapse whitespace

Returns:

  • (String) —

    Normalized text



37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 37

def normalize(text, options = {})
  return "" if text.nil?

  defaults = {
    downcase: true,
    strip_punct: false,
    collapse_ws: true
  }
  opts = defaults.merge(options)

  result = text.dup

  # Strip whitespace
  result = result.strip

  # Collapse multiple whitespace
  result = result.gsub(/\s+/, " ") if opts[:collapse_ws]

  # Downcase
  result = result.downcase if opts[:downcase]

  # Strip punctuation
  result = strip_punctuation(result) if opts[:strip_punct]

  result
end

#normalize_quotes(text) ⇒ String

Normalize quotes to standard ASCII.

Parameters:

  • text (String) —

    Text with quotes

Returns:

  • (String) —

    Text with normalized quotes



105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 105

def normalize_quotes(text)
  # Left double quote to straight
  text = text.gsub(/[\u201C\u201D]/, '"')
  # Right double quote to straight
  text = text.gsub(/[\u2018\u2019]/, "'")
  # Backticks to quotes
  text = text.gsub('`', "'")
  # Other quote variants
  text = text.gsub("\u00AB", '"')  # Left-pointing double angle
  text = text.gsub("\u00BB", '"')  # Right-pointing double angle
  text = text.gsub("\u2039", "'")  # Single left-pointing
  text.gsub("\u203A", "'") # Single right-pointing
end

#normalize_whitespace(text) ⇒ String

Normalize whitespace.

Parameters:

  • text (String) —

    Text with irregular whitespace

Returns:

  • (String) —

    Text with normalized whitespace



123
124
125
126
127
128
129
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 123

def normalize_whitespace(text)
  text
    .gsub(/[\u00A0\u202F\u205F]/, " ") # Various space chars
    .gsub(/[\u2000-\u200B]/, " ") # Various space chars
    .gsub(/\s+/, " ") # Collapse multiple spaces
    .strip
end

#normalize_word(word) ⇒ String

Normalize a word.

Parameters:

  • word (String) —

    Word to normalize

Returns:

  • (String) —

    Normalized word



68
69
70
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 68

def normalize_word(word)
  normalize(word)
end

#normalized_eql?(str1, str2) ⇒ Boolean

Check if two normalized strings are equal.

Parameters:

  • str1 (String) —

    First string

  • str2 (String) —

    Second string

Returns:

  • (Boolean) —

    True if equal after normalization



77
78
79
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 77

def normalized_eql?(str1, str2)
  normalize(str1) == normalize(str2)
end

#remove_accents(text) ⇒ String

Remove accents from characters.

Parameters:

  • text (String) —

    Text with accents

Returns:

  • (String) —

    Text without accents



93
94
95
96
97
98
99
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 93

def remove_accents(text)
  # Unicode normalization form D (decompose)
  normalized = text.unicode_normalize(:nfd)

  # Remove combining diacritical marks
  normalized.gsub(/[\u0300-\u036F]/, "")
end

#strip_punctuation(text) ⇒ String

Strip punctuation from text.

Parameters:

  • text (String) —

    Text to strip

Returns:

  • (String) —

    Text without punctuation



85
86
87
 
# File 'lib/kotoshu/language/normalizer/base.rb', line 85

def strip_punctuation(text)
  text.gsub(/[^\p{L}\p{N}\s]/, "")
end