Module: NEU::MODS::TextNormalizer

Defined in:
lib/neu/mods/canonicalize.rb

Overview

Normalises curator-authored freetext on the way into the JSON access copy (and Solr); the XML preservation copy stays untouched. Ported from Atlas's TextNormalizer (which carries DRS v1 prior art) so the gem reproduces Atlas's projection byte-for-byte.

IMPORTANT: every character-class regex is built programmatically from codepoint lists via format('\\u%04X', cp), so this source file stays pure ASCII -- no literal smart-quotes, dashes, or (critically) raw control bytes land on disk. Keep it that way.

Pipeline: force UTF-8 + scrub invalid bytes; NFC; map Unicode dashes to '-' (swung-dash to '~'); transliterate the General Punctuation block (smart quotes, ellipsis, etc.) to ASCII; map the separator controls to a newline and drop what is invisible (the soft hyphen, the rest of C0/C1, keeping tab/newline); collapse horizontal-whitespace runs to one space; for paragraph fields, collapse 2+ newlines to exactly two; strip.

.normalize(str)            -- single-line fields (newlines -> spaces)
.normalize_paragraphs(str) -- fields that may carry paragraph breaks
                            (abstract, accessCondition)

Constant Summary collapse

DASH_CODEPOINTS =

NOTE: U+2053 (swung dash) is intentionally excluded from dashes -- it is named "dash" but conventionally maps to ASCII '~', not '-' (V1 prior art).

[
  0x002D, 0x058A, 0x05BE, 0x1400, 0x1806,
  0x2010, 0x2011, 0x2012, 0x2013, 0x2014, 0x2015,
  0x2043, 0x207B, 0x208B, 0x2212,
  0x2E17, 0x2E1A, 0x2E3A, 0x2E3B, 0x2E40,
  0x301C, 0x3030, 0x30A0, 0xFE31, 0xFE32, 0xFE58,
  0xFE63, 0xFF0D
].freeze
DASH_RE = 
char_class(DASH_CODEPOINTS).freeze
SWUNG_DASH_RE = 
Regexp.new(format('\\u%04X', 0x2053)).freeze
SOFT_HYPHEN_RE =

U+00AD is a hint about where a word may break, not a dash: it renders as nothing, and Solr discards it, so "co<00AD>operation" already matches a search for "cooperation". Mapping it to an ASCII hyphen instead would index "co" and "operation" as two tokens and lose the word, so it is dropped and kept out of DASH_CODEPOINTS.

Regexp.new(format('\\u%04X', 0x00AD)).freeze
SEPARATOR_CONTROL_CODEPOINTS =

U+000B (vertical tab) and U+000C (form feed) separate words rather than meaning nothing: Word writes a manual line break as U+000B and a page break as U+000C. They map to a newline, because deleting one runs the words either side of it together -- normalize_paragraphs then reads that newline as the soft wrap the line break was, and normalize turns it into a space.

[0x000B, 0x000C].freeze
SEPARATOR_CONTROL_RE = 
char_class(SEPARATOR_CONTROL_CODEPOINTS).freeze
CONTROL_CODEPOINTS =

C0 (U+0000..U+0008, U+000D..U+001F) and C1 (U+007F..U+009F) -- what is left once the separators above are accounted for, and none of it carries meaning in curator text. U+0009 (tab) and U+000A (newline) are preserved. U+000D is not: dropping it reduces a CRLF line ending to the single newline it stands for.

((0x0000..0x0008).to_a + (0x000D..0x001F).to_a + (0x007F..0x009F).to_a).freeze
CONTROL_RE = 
char_class(CONTROL_CODEPOINTS).freeze
HORIZONTAL_WS_CODEPOINTS = 
[
  0x0009, 0x00A0, 0x1680,
  0x2000, 0x2001, 0x2002, 0x2003, 0x2004, 0x2005, 0x2006,
  0x2007, 0x2008, 0x2009, 0x200A, 0x202F, 0x205F, 0x3000
].freeze
HORIZONTAL_WS_RE =

Leading literal space included in the class (the " " prefix); + so a run of horizontal whitespace collapses to a single space.

Regexp.new("#{char_class(HORIZONTAL_WS_CODEPOINTS, prefix: " ").source}+").freeze
PARAGRAPH_RUN_RE = 
/\n{2,}/
GENERAL_PUNCTUATION =

General Punctuation block (U+2000..U+206F). Codepoints not listed pass through unchanged. Empty-string values deliberately drop invisible/bidi/ format marks so they cannot leak into the access copy.

{
  0x2000 => " ", 0x2001 => " ", 0x2002 => " ", 0x2003 => " ",
  0x2004 => " ", 0x2005 => " ", 0x2006 => " ", 0x2007 => " ",
  0x2008 => " ", 0x2009 => " ", 0x200A => " ",
  0x200B => "",  0x200C => "",  0x200D => "",
  0x200E => "",  0x200F => "",
  0x2018 => "'", 0x2019 => "'", 0x201A => ",", 0x201B => "'",
  0x201C => '"', 0x201D => '"', 0x201E => '"', 0x201F => '"',
  0x2020 => "+", 0x2021 => "+",
  0x2022 => "*", 0x2023 => "*", 0x2024 => ".", 0x2025 => "..",
  0x2026 => "...",
  0x2028 => "\n", 0x2029 => "\n\n",
  0x202A => "",  0x202B => "", 0x202C => "", 0x202D => "",
  0x202E => "",  0x202F => " ",
  0x2030 => "%", 0x2032 => "'", 0x2033 => '"', 0x2035 => "'",
  0x2036 => '"',
  0x2039 => "<", 0x203A => ">", 0x203C => "!!", 0x203D => "?",
  0x2044 => "/", 0x2052 => "%",
  0x205F => " ", 0x2060 => "", 0x2061 => "", 0x2062 => "",
  0x2063 => "",  0x2064 => "",
  0x206A => "",  0x206B => "", 0x206C => "", 0x206D => "",
  0x206E => "",  0x206F => ""
}.transform_keys { |cp| [cp].pack("U") }.freeze
GENERAL_PUNCTUATION_RE = 
Regexp.new("[#{format('\\u%04X-\\u%04X', 0x2000, 0x206F)}]").freeze

Class Method Summary collapse

Class Method Details

.base_normalize(str) ⇒ Object 



153
154
155
156
157
158
159
160
161
162
163
 
# File 'lib/neu/mods/canonicalize.rb', line 153

def base_normalize(str)
  s = str.dup.force_encoding("UTF-8")
  s = s.scrub("")
  s = s.unicode_normalize(:nfc)
  s = s.gsub(DASH_RE, "-")
  s = s.gsub(SWUNG_DASH_RE, "~")
  s = s.gsub(GENERAL_PUNCTUATION_RE) { |c| GENERAL_PUNCTUATION.fetch(c, c) }
  s = s.gsub(SEPARATOR_CONTROL_RE, "\n")
  s = s.gsub(SOFT_HYPHEN_RE, "")
  s.gsub(CONTROL_RE, "")
end

.char_class(codepoints, prefix: "") ⇒ Object

Build a character-class Regexp from an array of integer codepoints, as \uXXXX escapes (keeps this source ASCII).



54
55
56
 
# File 'lib/neu/mods/canonicalize.rb', line 54

def self.char_class(codepoints, prefix: "")
  Regexp.new("[#{prefix}#{codepoints.map { |cp| format('\\u%04X', cp) }.join}]")
end

.normalize(str) ⇒ Object 



135
136
137
138
139
140
141
 
# File 'lib/neu/mods/canonicalize.rb', line 135

def normalize(str)
  return "" if str.nil?

  s = base_normalize(str.to_s)
  s = s.tr("\n", " ")
  s.gsub(HORIZONTAL_WS_RE, " ").strip
end

.normalize_paragraphs(str) ⇒ Object 



143
144
145
146
147
148
149
150
151
 
# File 'lib/neu/mods/canonicalize.rb', line 143

def normalize_paragraphs(str)
  return "" if str.nil?

  s = base_normalize(str.to_s)
  s = s.gsub(HORIZONTAL_WS_RE, " ")
  s = s.gsub(/ *\n */, "\n")
  s.split(PARAGRAPH_RUN_RE).map { |p| p.tr("\n", " ").strip }
                           .reject(&:empty?).join("\n\n")
end