Module: JLPT::Preprocessor

Defined in:
lib/jlpt/core/preprocessor.rb

Overview

Text Preprocessor and Unicode Normalizer.

Cleans input text by stripping HTML, Markdown, URLs, and code blocks, normalizes Unicode NFKC characters (Half-width/Full-width Kana), and segments Japanese text into sentences.

Constant Summary collapse

HTML_TAG_REGEX = 
/<[^>]*>/.freeze
/\[([^\]]+)\]\([^)]+\)/.freeze
MARKDOWN_HEADER_CODE_REGEX = 
/^#+\s*|`[^`]+`|```[\s\S]*?```/.freeze
URL_REGEX = 
%r{https?://\S+}.freeze
SENTENCE_SCAN_REGEX = 
/[^。！？!?\n]+[。！？!?\n]?/.freeze

Class Method Summary collapse

Class Method Details

.clean(text) ⇒ String

Clean text by removing HTML, Markdown, and URLs

Parameters:

  • text (String) —

    Japanese raw text

Returns:

  • (String) —

    cleaned and sanitized text



22
23
24
25
26
27
28
29
30
31
 
# File 'lib/jlpt/core/preprocessor.rb', line 22

def clean(text)
  return '' if text.nil? || text.to_s.strip.empty?

  str = text.to_s.dup
  str.gsub!(HTML_TAG_REGEX, '')
  str.gsub!(URL_REGEX, '')
  str.gsub!(MARKDOWN_LINK_REGEX, '\1')
  str.gsub!(MARKDOWN_HEADER_CODE_REGEX, '')
  str.unicode_normalize(:nfkc).strip
end

.sentences(text) ⇒ Array<String>

Segment Japanese text into clean sentence arrays

Parameters:

  • text (String) —

    Japanese text

Returns:

  • (Array<String>) —

    array of non-empty sentences



37
38
39
40
41
42
43
44
45
 
# File 'lib/jlpt/core/preprocessor.rb', line 37

def sentences(text)
  cleaned = clean(text)
  return [] if cleaned.empty?

  matches = cleaned.scan(SENTENCE_SCAN_REGEX)
  result = matches.map(&:strip).reject(&:empty?)

  result.empty? ? [cleaned] : result
end