Detergent
Detergent scrubs junk out of web pages. Give it an HTML document and it removes ads, navigation, scripts, sidebars, and other cruft, then extracts the main article content.
Installation
Add to your Gemfile:
gem "detergent"
Usage
require "detergent"
cleaner = Detergent::Cleaner.new
# Get a cleaned, self-contained HTML document:
clean_html = cleaner.clean(dirty_html)
# Or just the title:
title = cleaner.title(dirty_html)
# Or the title and the extracted content node (a Nokogiri node):
title, content = cleaner.cleaned_html(dirty_html)
How it works
- A first pass prunes obvious junk (scripts, styles, iframes, nav, headers, footers, hidden elements).
- Each remaining node is scored on how likely it is to be the main content (paragraph count, text length, link density, media, suspicious class names, etc.).
- The highest-scoring node is located and walked up to an appropriate semantic container.
- A second pass cleans that content of empty and suspect elements.