Class: Detergent::Inspector

Inherits:
Object
  • Object
show all
Defined in:
lib/detergent/inspector.rb

Overview

Instruments a full extraction run and reports every decision made along the way: what the first pass removed, how the tree scored (with per-node score breakdowns), whether the best candidate cleared the minimum-score threshold, which content root was chosen, and what the second pass removed.

puts Detergent::Inspector.new.analyze(html)

Also available from the command line:

detergent --format debug page.html

Defined Under Namespace

Classes: Report, ScoredNode

Constant Summary collapse

TOP_NODE_COUNT = 
10
BREAKDOWN_COUNT = 
3

Instance Method Summary collapse

Instance Method Details

#analyze(html) ⇒ Object 



86
87
88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/detergent/inspector.rb', line 86

def analyze(html)
  @report = Report.new

  title, content = Cleaner.new(observer: self).extract(html)
  @report.title = title

  if content
    @report.content_root = descriptor(content)
    @report.content_root_size = content.to_s.length
  end

  @report
end

#content_pruned(body, scorer) ⇒ Object

Called after the first pass, before content location, with the same scorer the locator will use — so the report matches its decisions.



108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/detergent/inspector.rb', line 108

def content_pruned(body, scorer)
  scored = ([body] + body.xpath(".//*").to_a)
           .map { |node| [scorer.score(node), node] }
           .sort_by { |score, _node| -score }
           .first(TOP_NODE_COUNT)

  # Materialize descriptors/previews now; the tree mutates after this.
  @report.top_nodes = scored.each_with_index.map do |(score, node), index|
    breakdown = index < BREAKDOWN_COUNT ? scorer.explain(node) : nil
    ScoredNode.new(score, descriptor(node), preview(node), breakdown)
  end

  @report.best_score = scored.first&.first || 0
end

#extraction_strategy(strategy) ⇒ Object 



123
124
125
 
# File 'lib/detergent/inspector.rb', line 123

def extraction_strategy(strategy)
  @report.strategy = strategy
end

#node_removed(node, pass:) ⇒ Object

-- Cleaner observer callbacks --



102
103
104
 
# File 'lib/detergent/inspector.rb', line 102

def node_removed(node, pass:)
  @report.removals[pass] << node.name.downcase
end