Class: Html2rss::AutoSource::Cleanup

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source/cleanup.rb

Overview

Cleanup is responsible for cleaning up the extracted articles. :reek:MissingSafeMethod { enabled: false } It applies various strategies to filter and refine the article list.

Constant Summary collapse

DEFAULT_CONFIG =

Default cleanup behavior for auto-sourced article lists.

{
  keep_different_domain: false
}.freeze
MIN_WORDS =

Minimum alphanumeric word count for present titles.

3
VALID_SCHEMES =

Allowed URL schemes for article filtering.

%w[http https].to_set.freeze
CREDIT_TITLE =

Credit-agency-only or photo-credit titles (not headlines).

%r{
  \A(?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn)
    (?:\s*/\s*(?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn))*\z
  |
  \A(?:Image|Photo|Credit)\s*[:|]?\s*
    (?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn)\b
}ix
CMS_TOKEN_TITLE =

Dotted / methode CMS tokens mistaken for titles.

/\A(?:lucy\.\w[\w.-]*|methode[-.][\w.-]+)\z/i
SLUG_TITLE =

Raw URL slug / token clusters (hyphen or underscore, no natural phrasing).

/\A\p{Alnum}+(?:[-_]\p{Alnum}+){2,}\z/
DATE_PREFIX_TITLE =

Date-prefix path tokens, raw or titleized ("2026 08 16 …", "2026-08-16-…").

/\A\d{4}(?:[\s.-]+\d{1,2}){2}\b/
TITLEIZED_PATH_TITLE =

Titleized path ending in a long numeric CMS id.

/\A(?:\d+|\p{Lu}[\p{L}\p{M}]*)(?:\s+(?:\d+|\p{Lu}[\p{L}\p{M}]*))*\s+\d{6,}\z/
TEMPLATE_TITLE =

Template / placeholder tokens mistaken for titles.

/(\{\{[^}]+\}\}|%\{\w+\})/

Class Method Summary collapse

Class Method Details

.call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) ⇒ Array<Article>

Returns cleaned article list.

Parameters:

  • articles (Array<Article>) —

    extracted article candidates

  • url (Html2rss::Url) —

    feed source URL used for same-host filtering

  • keep_different_domain (Boolean) (defaults to: DEFAULT_CONFIG.fetch(:keep_different_domain)) —

    whether to keep off-domain entries

Returns:

  • (Array<Article>) —

    cleaned article list



50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/html2rss/auto_source/cleanup.rb', line 50

def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain))
  Log.debug "Cleanup: start with #{articles.size} articles"

  articles.select!(&:valid?)

  deduplicate_by_url!(articles)
  keep_only_http_urls!(articles)
  reject_self_links!(articles, url)
  reject_different_domain!(articles, url) unless keep_different_domain
  reject_low_quality_titles!(articles)

  Log.debug "Cleanup: end with #{articles.size} articles"
  articles
end