Class: Detergent::LinkListExtractor

Inherits:

Object

• Object
• Detergent::LinkListExtractor

Defined in:

lib/detergent/link_list_extractor.rb

Overview

Fallback extractor for index pages: link aggregators, front pages, and directories (e.g. the Hacker News front page) where the "content" is a list of links to other pages. The article locator correctly rejects these — link density is its strongest navigation signal — so when it finds nothing, this extractor looks for a substantial collection of title-like links and returns them as a synthetic list node that flows through the normal cleaning and rendering pipeline.

Constant Summary

MIN_LINK_TEXT_LENGTH =

Links with less text than this are treated as chrome ("login", "hide", "42 comments"), not titles.

15

MIN_LINKS =

Fewer title-like links than this isn't an index page.

10

Instance Method Summary

• #extract(body) ⇒ Object

  Returns a synthetic

  of the page's title-like links, or nil if the page doesn't look like a link index.

Instance Method Details

#extract(body) ⇒ Object

Returns a synthetic

of the page's title-like links, or nil if the page doesn't look like a link index.

# File 'lib/detergent/link_list_extractor.rb', line 21

def extract(body)
  links = title_like_links(body)
  return nil if links.length < MIN_LINKS

  build_list(body.document, links)
end