Class: Detergent::Cleaner

Inherits:
Object
  • Object
show all
Defined in:
lib/detergent/cleaner.rb

Overview

Orchestrates cleaning: prunes obvious junk, locates the main content, scrubs it, and renders the result back out as a standalone document.

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Cleaner

Returns a new instance of Cleaner.



7
8
9
10
 
# File 'lib/detergent/cleaner.rb', line 7

def initialize
  @obvious_junk_matcher = Matchers::ObviousJunkMatcher.new
  @removable_node_matcher = Matchers::RemovableNodeMatcher.new
end

Instance Method Details

#clean(html) ⇒ Object

Returns a complete, standalone HTML document containing only the page's title and its extracted main content.



14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/detergent/cleaner.rb', line 14

def clean(html)
  title, content = extract(html)

  # The content root is usually an inner element (article, main, div),
  # so wrap it in a body tag unless it already is one.
  body = if content.nil? || content.name.downcase == "body"
    content.to_s
  else
    "<body>#{content}</body>"
  end

  <<~HTML
    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>#{CGI.escapeHTML(title.to_s)}</title>
      </head>
      #{body}
    </html>
  HTML
end

#extract(html) ⇒ Object

Returns [title, content]: the page title and the cleaned Nokogiri node containing the main content (nil if none was found).



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/detergent/cleaner.rb', line 40

def extract(html)
  doc = Nokogiri::HTML5(html)
  title = extract_title(doc)

  body = doc.at('body')
  content = nil

  if body
    # First remove the most egregious crap, like obvious ads, etc.
    prune(node: body, matcher: @obvious_junk_matcher)

    # Score caches are only valid for a single parse of a single
    # document, so the scorer and locator are built per extract.
    content = ContentLocator.new(NodeScorer.new).locate(body)

    # Apply second-pass cleaning to the content
    if content
      clean_node(content)
      strip_junk_attributes(content)
    end
  end

  [title, content]
end

#markdown(html) ⇒ Object

Returns the extracted main content as Markdown.



66
67
68
69
 
# File 'lib/detergent/cleaner.rb', line 66

def markdown(html)
  _, content = extract(html)
  content ? MarkdownRenderer.new.render(content) : ""
end

#text(html) ⇒ Object

Returns the extracted main content as plain text.



72
73
74
75
 
# File 'lib/detergent/cleaner.rb', line 72

def text(html)
  _, content = extract(html)
  content ? TextRenderer.new.render(content) : ""
end

#title(html) ⇒ Object 



77
78
79
 
# File 'lib/detergent/cleaner.rb', line 77

def title(html)
  extract_title(Nokogiri::HTML5(html))
end