Module: Html2rss::Html::ArticleRules::Description

Defined in:
lib/html2rss/html/article_rules/description.rb

Overview

Leftover visible-text keep/drop for article descriptions. Split once on block newlines; Date and Category consume the same keepers.

Constant Summary collapse

CTA =

CTA lines dropped by whole-line equality (case-insensitive).

Set['read more', 'learn more'].freeze
TYPE_CHIPS =

Type-chip lines dropped when the whole leftover line is one of these.

Set['news article', 'press release', 'news'].freeze
SECTION_NAMES =

Listing section names that are chrome, not a dek.

Set['press releases'].freeze
MAX_DATE_CHARS =

Skip date detection / DateTime.parse above this length.

128
CHIP_SEPARATORS =

Optional type-chip / clock separators (ASCII/en/em dash, pipe, bullet).

[' - ', ' – ', ' — ', ' | ', ' • '].freeze
CLOCK_ZONE =

Trailing clock plus optional AM/PM and one timezone/offset token (IANA, GMT/UTC offset, or abbrev).

%r{
  \A
  \d{1,2}:\d{2}(?::\d{2})?
  (?:\s*(?:AM|PM))?
  (?:\s+(?:[A-Za-z]+(?:/[A-Za-z_]+)+|(?:GMT|UTC)[+-]\d{1,2}(?::\d{2})?|[A-Za-z]{2,5}))?
  \z
}ix
DATE_SHAPES =

Whole-line date shapes after normalize (ISO, numeric, day-month, month-day).

[
  /\A\d{4}-\d{1,2}-\d{1,2}(?:[T ]\d{1,2}:\d{2}(?::\d{2})?(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?)?\z/i,
  %r{\A\d{1,2}[./-]\d{1,2}[./-]\d{2,4}(?: \d{1,2}:\d{2}(?::\d{2})?)?\z},
  /\A\d{1,2}\.? \p{L}{3,9}\.?\s*,?\s*\d{4}(?: \d{1,2}:\d{2}(?::\d{2})?)?\z/i,
  /\A\p{L}{3,9}\.? \d{1,2}\s*,?\s*\d{4}(?: \d{1,2}:\d{2}(?::\d{2})?)?\z/i
].freeze

Class Method Summary collapse

Class Method Details

.date_core(line) ⇒ String

Parameters:

  • line (String, nil)

Returns:

  • (String) 


78
79
80
81
82
83
84
85
 
# File 'lib/html2rss/html/article_rules/description.rb', line 78

def date_core(line)
  normalized = normalize(line)
  CHIP_SEPARATORS.each do |sep|
    peeled = peel_separator(normalized, sep)
    return peeled if peeled
  end
  normalized
end

.date_shaped?(line) ⇒ Boolean

Parameters:

  • line (String, nil)

Returns:

  • (Boolean) 


68
69
70
71
72
73
 
# File 'lib/html2rss/html/article_rules/description.rb', line 68

def date_shaped?(line)
  core = date_core(line)
  return false if core.empty? || core.length > MAX_DATE_CHARS

  DATE_SHAPES.any? { |shape| core.match?(shape) }
end

.from_lines(lines, title: nil) ⇒ String?

Parameters:

  • lines (Array<String>, nil)
  • title (String, nil) (defaults to: nil)

Returns:

  • (String, nil) 


48
49
50
51
 
# File 'lib/html2rss/html/article_rules/description.rb', line 48

def from_lines(lines, title: nil)
  kept = Array(lines).select { |line| keep?(line, title:) }
  kept.empty? ? nil : kept.join("\n")
end

.keep?(line, title: nil, type_chips: true) ⇒ Boolean

Parameters:

  • line (String, nil)
  • title (String, nil) (defaults to: nil)
  • type_chips (Boolean) (defaults to: true) —

    whether whole-line type chips are leftover chrome

Returns:

  • (Boolean) 


58
59
60
61
62
63
 
# File 'lib/html2rss/html/article_rules/description.rb', line 58

def keep?(line, title: nil, type_chips: true)
  normalized = normalize(line)
  return false if normalized.empty?

  !chrome?(normalized, title:, type_chips:)
end

.lines_from(text) ⇒ Array<String>

Parameters:

  • text (String, nil)

Returns:

  • (Array<String>) 


40
41
42
 
# File 'lib/html2rss/html/article_rules/description.rb', line 40

def lines_from(text)
  text.to_s.split(/\n+/).map { |line| normalize(line) }.reject(&:empty?)
end