Class: Detergent::ContentLocator

Inherits:
Object
  • Object
show all
Defined in:
lib/detergent/content_locator.rb

Overview

Locates the main content node within a document's body: finds the highest-scoring node in the tree, then walks up its ancestors to find the best root container for the content.

Constant Summary collapse

SEMANTIC_CONTAINERS = 
%w[article main section].freeze
PARENT_PROMOTION_RATIO =

A parent must beat the best root found so far by this ratio to be promoted — not just score marginally higher from including the child.

1.3
MINIMUM_CONTENT_SCORE =

If even the best-scoring node falls below this, the page has no main content to extract. Pages made entirely of links (index pages, link aggregators like the Hacker News front page) score at or near zero everywhere, and without a floor a degenerate winner gets elected — e.g. a logo cell that picked up a few points for containing an image.

25

Instance Method Summary collapse

Constructor Details

#initialize(scorer) ⇒ ContentLocator

Returns a new instance of ContentLocator.



21
22
23
 
# File 'lib/detergent/content_locator.rb', line 21

def initialize(scorer)
  @scorer = scorer
end

Instance Method Details

#locate(body) ⇒ Object

Returns the main content root node, or nil if the page doesn't appear to have main content at all.



27
28
29
30
31
32
33
 
# File 'lib/detergent/content_locator.rb', line 27

def locate(body)
  highest_scoring = find_highest_scoring_node(body)
  return nil unless highest_scoring
  return nil if score(highest_scoring) < MINIMUM_CONTENT_SCORE

  find_content_root(highest_scoring)
end