Class: Html2rss::AutoSource::Cleanup

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source/cleanup.rb

Overview

Cleanup is responsible for cleaning up the extracted articles. :reek:MissingSafeMethod { enabled: false } It applies various strategies to filter and refine the article list. Sole producer of admission drop tallies for Status.

Defined Under Namespace

Classes: Result

Constant Summary collapse

DEFAULT_CONFIG =

Default cleanup behavior for auto-sourced article lists.

{
  keep_different_domain: false
}.freeze
MIN_WORDS =

Minimum alphanumeric word count for present titles.

3
VALID_SCHEMES =

Allowed URL schemes for article filtering.

%w[http https].to_set.freeze

Class Method Summary collapse

Class Method Details

.call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) ⇒ Result

Returns cleaned articles and frozen drop tallies.

Parameters:

  • articles (Array<Article>) —

    extracted article candidates

  • url (Html2rss::Url) —

    feed source URL used for same-host filtering

  • keep_different_domain (Boolean) (defaults to: DEFAULT_CONFIG.fetch(:keep_different_domain)) —

    whether to keep off-domain entries

Returns:

  • (Result) —

    cleaned articles and frozen drop tallies



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/html2rss/auto_source/cleanup.rb', line 62

def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) # rubocop:disable Metrics/MethodLength -- ordered reject pipeline
  Log.debug "Cleanup: start with #{articles.size} articles"
  tallies = Hash.new(0)

  reject_invalid!(articles, tallies)
  deduplicate_by_url!(articles, tallies)
  keep_only_http_urls!(articles, tallies)
  reject_self_links!(articles, url, tallies)
  reject_different_domain!(articles, url, tallies) unless keep_different_domain
  reject_excluded_destinations!(articles, tallies)
  reject_low_quality_titles!(articles, tallies)

  Log.debug "Cleanup: end with #{articles.size} articles"
  Result.new(articles:, drop_tallies: tallies.freeze)
end

.junk_reason(title) ⇒ Symbol?

First matching junk reason for a title, or nil when the title is acceptable.

Parameters:

  • title (String, nil) —

    candidate title text

Returns:

  • (Symbol, nil) 


82
83
84
85
86
87
88
89
 
# File 'lib/html2rss/auto_source/cleanup.rb', line 82

def junk_reason(title)
  return if title.nil?

  normalized = normalize_title(title)
  return if normalized.empty?

  JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
end