Class: Ask::WebFetch::ContentFilter

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/web_fetch/content_filter.rb

Overview

Density-based content pruning, ported from crawl4ai's PruningContentFilter (Apache-2.0, https://github.com/unclecode/crawl4ai).

Scores every element in the page by how much of it is real text versus markup and links, then removes elements scoring below a threshold. Where a keyword scraper fails ("remove anything with 'nav' in the class"), this keeps content that scores well regardless of its label and drops link-farms that happen to dodge the keywords.

Two deliberate deviations from crawl4ai, both adding tags crawl4ai's own included_tags list marks as content but whose tag_weights map left out (they defaulted to 0.5 and could be pruned despite being article content):

joins
/
at the top tier, and the table/pre/code family gets content-worthy weights. Third: the class/id chrome penalty actually subtracts from the score — crawl4ai floors it at zero (max(0, ...)), which makes its negative-patterns metric inert. Fourth: is excluded outright — chart text would otherwise leak into the markdown as concatenated axis labels ("01M2M3M", "10Apr15Apr").

Constant Summary collapse

EXCLUDED_TAGS =

Structural boilerplate, removed before scoring ever runs. The preserve_* whitelist cannot save these.

%w[nav footer header aside script style form iframe noscript svg].freeze
NEGATIVE_PATTERNS =

Class/id fragments that mark non-content chrome; matching one knocks 0.5 off the node's score.

/nav|footer|header|sidebar|ads|comment|promo|advert|social|share/i.freeze
TAG_WEIGHTS =

Semantic tag weights: how likely a tag is to carry article content.

{
  'div' => 0.5, 'p' => 1.0, 'article' => 1.5, 'section' => 1.0, 'main' => 1.4,
  'span' => 0.3, 'li' => 0.5, 'ul' => 0.5, 'ol' => 0.5,
  'h1' => 1.2, 'h2' => 1.1, 'h3' => 1.0, 'h4' => 0.9, 'h5' => 0.8, 'h6' => 0.7,
  'table' => 1.0, 'tr' => 0.8, 'td' => 0.8, 'th' => 0.8,
  'pre' => 1.0, 'code' => 0.9, 'blockquote' => 1.0
}.freeze
TAG_IMPORTANCE =

Used only by the dynamic threshold, which loosens the bar for tags that usually carry content.

{
  'article' => 1.5, 'main' => 1.4, 'section' => 1.3, 'p' => 1.2,
  'h1' => 1.4, 'h2' => 1.3, 'h3' => 1.2, 'div' => 0.7, 'span' => 0.6
}.freeze
METRIC_WEIGHTS = 
{
  text_density: 0.4, link_density: 0.2, tag_weight: 0.2,
  class_id_weight: 0.1, text_length: 0.1
}.freeze
DEFAULT_THRESHOLD = 
0.48

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(threshold: DEFAULT_THRESHOLD, threshold_type: :fixed, min_word_threshold: nil, preserve_classes: [], preserve_tags: []) ⇒ ContentFilter

threshold

score below this removes the element

threshold_type

:fixed or :dynamic — dynamic loosens the bar for important tags and text-heavy, link-light nodes

min_word_threshold

elements with fewer words are removed outright

preserve_classes

class names never pruned, regardless of score

preserve_tags

tag names never pruned, regardless of score



71
72
73
74
75
76
77
78
 
# File 'lib/ask/web_fetch/content_filter.rb', line 71

def initialize(threshold: DEFAULT_THRESHOLD, threshold_type: :fixed,
               min_word_threshold: nil, preserve_classes: [], preserve_tags: [])
  @threshold = threshold
  @threshold_type = threshold_type.to_sym
  @min_word_threshold = min_word_threshold
  @preserve_classes = preserve_classes
  @preserve_tags = preserve_tags
end

Class Method Details

.default ⇒ Object

The filter the backends use by default: the adaptive threshold, which loosens the bar for content-carrying tags and text-heavy nodes and tightens it for link-heavy ones — catching the classic sidebar-of-links that the fixed bar lets through.



61
62
63
 
# File 'lib/ask/web_fetch/content_filter.rb', line 61

def self.default
  new(threshold_type: :dynamic)
end

Instance Method Details

#filter_content(html) ⇒ Object

html -> [String] the surviving top-level blocks, as HTML fragments. Empty input yields an empty array; malformed HTML is parsed with Nokogiri's recover mode and never raises.



83
84
85
86
87
88
89
90
91
92
93
94
95
 
# File 'lib/ask/web_fetch/content_filter.rb', line 83

def filter_content(html)
  return [] if html.nil? || html.empty?

  doc = Nokogiri::HTML(html)
  body = doc.at_css('body') || doc.at_css('html')
  return [] unless body

  remove_comments(doc)
  body.css(EXCLUDED_TAGS.join(',')).each(&:remove)
  body.element_children.each { |child| prune(child) }

  body.element_children.select { |el| el.text.strip.length.positive? }.map(&:to_html)
end

#fit_html(html) ⇒ Object

html -> String, the surviving blocks wrapped in

s, ready for markdown conversion. Mirrors crawl4ai's generator, which wraps the filtered blocks before converting them.



100
101
102
 
# File 'lib/ask/web_fetch/content_filter.rb', line 100

def fit_html(html)
  filter_content(html).map { |block| "<div>#{block}</div>" }.join
end