Module: Detergent

Defined in:
lib/detergent.rb,
lib/detergent/cleaner.rb,
lib/detergent/version.rb,
lib/detergent/inspector.rb,
lib/detergent/node_scorer.rb,
lib/detergent/text_renderer.rb,
lib/detergent/content_locator.rb,
lib/detergent/markdown_renderer.rb,
lib/detergent/link_list_extractor.rb,
lib/detergent/matchers/obvious_junk_matcher.rb,
lib/detergent/matchers/removable_node_matcher.rb

Defined Under Namespace

Modules: Matchers Classes: Cleaner, ContentLocator, Inspector, LinkListExtractor, MarkdownRenderer, NodeScorer, TextRenderer

Constant Summary collapse

JUNK_TAGS =

Tags that never contain readable content.

%w[script style link iframe noscript].freeze
CHROME_TAGS =

Structural chrome that surrounds the content.

%w[nav header footer].freeze
FORM_TAGS =

Interactive elements that don't belong in cleaned output.

%w[form select textarea].freeze
MEDIA_TAGS =

Media elements that legitimately contain no text.

%w[img picture figure].freeze
VERSION = 
"2.4.0"

Class Method Summary collapse

Class Method Details

.clean(html) ⇒ Object

Returns a cleaned, standalone HTML document.



25
26
27
 
# File 'lib/detergent.rb', line 25

def self.clean(html)
  Cleaner.new.clean(html)
end

.display_none?(node) ⇒ Boolean

Returns:

  • (Boolean) 


19
20
21
22
 
# File 'lib/detergent.rb', line 19

def self.display_none?(node)
  style = node["style"].to_s.downcase
  style.include?("display:none") || style.include?("display: none")
end

.extract(html) ⇒ Object

Returns [title, content]: the page title and the cleaned Nokogiri node containing the main content (nil if none was found).



31
32
33
 
# File 'lib/detergent.rb', line 31

def self.extract(html)
  Cleaner.new.extract(html)
end

.markdown(html) ⇒ Object

Returns the extracted main content as Markdown.



36
37
38
 
# File 'lib/detergent.rb', line 36

def self.markdown(html)
  Cleaner.new.markdown(html)
end

.text(html) ⇒ Object

Returns the extracted main content as plain text.



41
42
43
 
# File 'lib/detergent.rb', line 41

def self.text(html)
  Cleaner.new.text(html)
end