Class: Detergent::Cleaner

Inherits:
Object
  • Object
show all
Defined in:
lib/detergent/cleaner.rb

Overview

Orchestrates cleaning: prunes obvious junk, locates the main content, scrubs it, and renders the result back out as a standalone document.

Instance Method Summary collapse

Constructor Details

#initialize(observer: nil) ⇒ Cleaner

observer (optional) receives instrumentation callbacks during extraction: node_removed(node, pass:), content_pruned(body, scorer), and extraction_strategy(strategy). Used by Inspector to build debug reports.



11
12
13
14
15
 
# File 'lib/detergent/cleaner.rb', line 11

def initialize(observer: nil)
  @observer = observer
  @obvious_junk_matcher = Matchers::ObviousJunkMatcher.new
  @removable_node_matcher = Matchers::RemovableNodeMatcher.new
end

Instance Method Details

#clean(html) ⇒ Object

Returns a complete, standalone HTML document containing only the page's title and its extracted main content.



19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/detergent/cleaner.rb', line 19

def clean(html)
  title, content = extract(html)

  # The content root is usually an inner element (article, main, div),
  # so wrap it in a body tag unless it already is one.
  body = if content.nil? || content.name.downcase == "body"
    content.to_s
  else
    "<body>#{content}</body>"
  end

  <<~HTML
    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>#{CGI.escapeHTML(title.to_s)}</title>
      </head>
      #{body}
    </html>
  HTML
end

#extract(html) ⇒ Object

Returns [title, content]: the page title and the cleaned Nokogiri node containing the main content (nil if none was found).



45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
 
# File 'lib/detergent/cleaner.rb', line 45

def extract(html)
  doc = Nokogiri::HTML5(html)
  title = extract_title(doc)

  body = doc.at('body')
  content = nil

  if body
    # First remove the most egregious crap, like obvious ads, etc.
    prune(node: body, matcher: @obvious_junk_matcher)

    # Score caches are only valid for a single parse of a single
    # document, so the scorer and locator are built per extract.
    scorer = NodeScorer.new
    @observer&.content_pruned(body, scorer)
    content = ContentLocator.new(scorer).locate(body)
    strategy = content ? :article : nil

    # No article-shaped content: the page may be a link index (front
    # page, aggregator), whose content is its list of links.
    if content.nil?
      content = LinkListExtractor.new.extract(body)
      strategy = :link_list if content
    end

    @observer&.extraction_strategy(strategy)

    # Apply second-pass cleaning to the content
    if content
      clean_node(content)
      strip_junk_attributes(content)
    end
  end

  [title, content]
end

#markdown(html) ⇒ Object

Returns the extracted main content as Markdown.



83
84
85
86
 
# File 'lib/detergent/cleaner.rb', line 83

def markdown(html)
  _, content = extract(html)
  content ? MarkdownRenderer.new.render(content) : ""
end

#text(html) ⇒ Object

Returns the extracted main content as plain text.



89
90
91
92
 
# File 'lib/detergent/cleaner.rb', line 89

def text(html)
  _, content = extract(html)
  content ? TextRenderer.new.render(content) : ""
end

#title(html) ⇒ Object 



94
95
96
 
# File 'lib/detergent/cleaner.rb', line 94

def title(html)
  extract_title(Nokogiri::HTML5(html))
end