Class: Detergent::NodeScorer

Inherits:
Object
  • Object
show all
Defined in:
lib/detergent/node_scorer.rb

Overview

Scores nodes on how likely they are to be the page's main content.

Descendant statistics are computed bottom-up, with each node's stats built from its children's already-computed stats, so scoring every node in a document is O(n) instead of a full-subtree scan per node. A scorer instance is scoped to a single parse of a single document; discard it once the tree is mutated.

Constant Summary collapse

ARTICLE_TAG_BONUS = 
100
MAIN_TAG_BONUS = 
50
MAIN_ROLE_BONUS = 
25
POINTS_PER_PARAGRAPH = 
5
CHARS_PER_TEXT_POINT = 
100
10
POINTS_PER_MEDIA = 
10
LONG_PARAGRAPH_BONUS = 
15
LONG_PARAGRAPH_LENGTH = 
100
POINTS_PER_BLOCKQUOTE = 
10
POINTS_PER_LIST = 
5
50
COMMENT_PENALTY = 
50
AD_PENALTY = 
70
SOCIAL_PENALTY = 
70

Instance Method Summary collapse

Constructor Details

#initialize ⇒ NodeScorer

Returns a new instance of NodeScorer.



28
29
30
31
 
# File 'lib/detergent/node_scorer.rb', line 28

def initialize
  @scores = {}.compare_by_identity
  @stats = {}.compare_by_identity
end

Instance Method Details

#explain(node) ⇒ Object

Returns the score as [label, points] pairs explaining where every point came from. Debugging aid used by Inspector.



44
45
46
47
48
 
# File 'lib/detergent/node_scorer.rb', line 44

def explain(node)
  return [] unless node.element?

  components_for(node)
end

#score(node) ⇒ Object 



33
34
35
36
37
38
39
40
 
# File 'lib/detergent/node_scorer.rb', line 33

def score(node)
  cached = @scores[node]
  return cached unless cached.nil?
  return 0 unless node.element?

  total = components_for(node).sum { |_label, points| points }
  @scores[node] = [total, 0].max # Don't return negative scores
end