Class: Html2rss::LinkDestination::PathClassifier

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/link_destination/path_classifier.rb

Overview

Classifies normalized destination path segments for scoring.

Constant Summary collapse

SOFT_UTILITY =

Soft utility segments excluded from high-confidence junk (still utility_path).

%w[
  for-you join member members membership newsletter newsletters
  plans plus premium pricing recommended
].to_set.freeze
SEGMENT_SETS =

Segment groups used to classify article, taxonomy, utility, and vanity routes. Utility = taxonomy ∪ chrome ∪ soft utility (do not re-list taxonomy tokens).

begin
  content = %w[
    article articles blog blogs changelog changelogs insight insights
    launch launches news post posts release releases story stories update updates
    artikel beitrag beitraege nachrichten neuigkeiten aktuelles
    articulo articulos noticia noticias entrada entradas publicacion publicaciones
    actualite actualites nouvelle nouvelles teaser teasers card cards
  ].to_set.freeze
  taxonomy = %w[
    category categories tag tags topic topics
    kategorie kategorien schlagwort schlagworte thema themen
    categoria categorias etiqueta etiquetas tema temas
    categorie etiquette etiquettes sujet sujets theme themes
  ].to_set.freeze
  vanity = %w[
    join membership plus premium pricing plans subscribe signup
    abonnieren abo suscribirse boletin s-abonner saboner
  ].to_set.freeze
  chrome = %w[
    about account archive archives author authors comment comments contact feedback
    help login logout notification notifications preference preferences profile register
    search settings share signup subscribe feed feeds comment-feed comments-feed privacy
    terms cookie cookies user users autor autoren archiv ueber-uns ueber ueberuns profil
    kontakt impressum suche hilfe anmelden registrieren konto registrierung anmeldung
    abonnieren abo datenschutz nutzungsbedingungen agb autores archivos sobre-nosotros
    sobre quienes-somos buscar busqueda ayuda entrar ingresar registrarse registro cuenta
    suscribirse boletin privacidad condiciones auteur auteurs a-propos apropos recherche
    rechercher aide connexion s-inscrire sinscrire inscription compte s-abonner saboner
    lettre-information confidentialite mentions-legales cgu menu sidebar widget social
    modal popup banner promo ad ads related recommendation recommendations pagination pager
    dating jobs job career careers deals deal shopping shop trading broker versicherung
    tierversicherung insurance vergleich comparison partnerboerse singleboerse krypto crypto
    casinos casino kreditkarten kreditkarte kredit echtgeld vpn games kaufberater leasing
  ].to_set.freeze
  utility = (taxonomy | chrome | SOFT_UTILITY).freeze
  {
    content:,
    utility:,
    high_confidence_junk: (utility - SOFT_UTILITY).freeze,
    taxonomy:,
    vanity:,
    deep_post_context: %w[press newsroom presse pressemitteilungen prensa].to_set.freeze
  }.freeze
end
YEARISH_SEGMENT =

Path segment that begins with a year-like publishing marker.

/\A\d{4,}[\w-]*\z/
POST_SLUG_SEGMENT =

Hyphenated slug shape common to article permalinks.

/\A[a-z0-9]+(?:-[a-z0-9]+){2,}\z/i
HOST_SHAPED_SEGMENT =

Multi-label host mistaken for a path segment (affiliate/outlink chrome). Final label must be alphabetic (TLD-like); exclude common file extensions.

/\A(?:www\.)?(?:[\w-]+\.)+[a-z]{2,24}\z/i
FILE_EXTENSION_SEGMENT =

Common file-extension suffixes excluded from host-shaped junk matching.

/\.(?:pdf|jpe?g|png|gif|svg|webp|css|js|mjs|html?|xml|json|mp4|webm|zip|gz)\z/i

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(segments) ⇒ PathClassifier

Returns a new instance of PathClassifier.

Parameters:

  • segments (Array<String>) —

    normalized URL path segments



72
73
74
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 72

def initialize(segments)
  @segments = segments
end

Instance Attribute Details

#segments ⇒ Object (readonly)

rubocop:disable Metrics/ClassLength



7
8
9
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 7

def segments
  @segments
end

Instance Method Details

#content_path? ⇒ Boolean

Returns true when the route has article-like path evidence.

Returns:

  • (Boolean) —

    true when the route has article-like path evidence



77
78
79
80
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 77

def content_path?
  @content_path ||= !leading_high_confidence_junk? &&
                    (SEGMENT_SETS[:content].intersect?(segments) || yearish_content_context?)
end

#deep_utility_context_route? ⇒ Boolean

Returns true when the leading segments are all utility chrome.

Returns:

  • (Boolean) —

    true when the leading segments are all utility chrome



126
127
128
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 126

def deep_utility_context_route?
  all_junk?(segments.size - 1)
end

#junk_path? ⇒ Boolean

Returns true when the route is shallow and contains high-confidence noise.

Returns:

  • (Boolean) —

    true when the route is shallow and contains high-confidence noise



131
132
133
134
135
136
137
138
139
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 131

def junk_path?
  return false if excluded_content_route?

  any_high_confidence_junk_segment? ||
    taxonomy_path? ||
    utility_only_route? ||
    deep_utility_context_route? ||
    shallow_high_confidence_route?
end

#shallow? ⇒ Boolean

Returns true when the route is too shallow to strongly indicate an article.

Returns:

  • (Boolean) —

    true when the route is too shallow to strongly indicate an article



98
99
100
101
102
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 98

def shallow?
  segment_count = segments.size

  segment_count <= 1 || (segment_count == 2 && high_confidence_junk_segment?(segments.last))
end

#shallow_high_confidence_route? ⇒ Boolean

Returns true when the route is shallow and contains high-confidence noise.

Returns:

  • (Boolean) —

    true when the route is shallow and contains high-confidence noise



117
118
119
120
121
122
123
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 117

def shallow_high_confidence_route?
  vanity_segments = SEGMENT_SETS.fetch(:vanity)

  shallow? && segments.any? do |segment|
    high_confidence_junk_segment?(segment) || vanity_segments.include?(segment)
  end
end

#strong_post_suffix? ⇒ Boolean

Returns true when the final path segment looks like a post slug.

Returns:

  • (Boolean) —

    true when the final path segment looks like a post slug



105
106
107
108
109
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 105

def strong_post_suffix?
  @strong_post_suffix ||= segments.any? &&
                          included_last_segment? &&
                          trusted_post_context?(segments.size - 1)
end

#taxonomy_path? ⇒ Boolean

Returns true when the route points at taxonomy/listing chrome.

Returns:

  • (Boolean) —

    true when the route points at taxonomy/listing chrome



93
94
95
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 93

def taxonomy_path?
  @taxonomy_path ||= SEGMENT_SETS[:taxonomy].intersect?(segments)
end

#utility_destination? ⇒ Boolean

Returns true when the route points at conversion or account chrome.

Returns:

  • (Boolean) —

    true when the route points at conversion or account chrome



142
143
144
145
146
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 142

def utility_destination?
  return false if excluded_content_route?

  vanity_path? || utility_route?
end

#utility_only_route? ⇒ Boolean

Returns true when every path segment is utility chrome.

Returns:

  • (Boolean) —

    true when every path segment is utility chrome



112
113
114
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 112

def utility_only_route?
  segments.all? { |segment| high_confidence_junk_segment?(segment) }
end

#utility_path? ⇒ Boolean

Returns true when the route includes utility/navigation evidence.

Returns:

  • (Boolean) —

    true when the route includes utility/navigation evidence



83
84
85
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 83

def utility_path?
  @utility_path ||= SEGMENT_SETS[:utility].intersect?(segments)
end

#vanity_path? ⇒ Boolean

Returns true when the route points at conversion or account chrome.

Returns:

  • (Boolean) —

    true when the route points at conversion or account chrome



88
89
90
 
# File 'lib/html2rss/link_destination/path_classifier.rb', line 88

def vanity_path?
  @vanity_path ||= SEGMENT_SETS[:vanity].intersect?(segments)
end