Class: Ask::WebFetch::NoiseFilter

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/web_fetch/noise_filter.rb

Overview

Strips decorative symbol noise from markdown: the long, letter-free, repetitive character streams pages render as animated backgrounds, marquees and section dividers — e.g. Hugging Face's storage page ships a "+ = · ( ~ @ # % & * ? / : ; < > [ ] { } | ^ $ !" stream as its page background. Runs on CONVERTED markdown, so every backend benefits: Local and Browser already convert through Markdown, and Jina and Crawl4AI hand the gem pre-converted markdown — the DOM-level ContentFilter never sees either case.

Conservative by design. A line is dropped only when ALL hold:

* it is long enough to matter (>= min_length characters)
* it contains no letters or digits at all
* it is repetitive — at least two distinct characters, with a
distinct/length ratio below max_entropy (a repeated stream,
not prose punctuation)
* it is not markdown structure: fenced or indented code, table
rows, headings, blockquotes, inline code, raw HTML, math

Short decorative fragments (an ASCII-art header like "*****"), single-character runs ("-----" dividers), and everything containing words survive. Lines whose only content is invisible characters (zero-width spaces, combining marks) are always dropped — they carry nothing. Blank lines are untouched.

Constant Summary collapse

DEFAULT_MIN_LENGTH =

Longest line that is never touched, whatever its contents.

32
DEFAULT_MAX_ENTROPY =

Highest distinct-chars/length ratio a line may have and still count as repetitive. Below this the line reads as a repeated stream; above it, as prose punctuation (kept).

0.3
INVISIBLE_RE =

Characters that never render: zero-width space/joiner and bidi controls, the BOM, and combining marks.

/[\u200B-\u200F\uFEFF\u2060\u00AD\p{Mn}]/.freeze
STRUCTURE_PREFIX_RE =

A line starting with one of these is structure, not noise: headings, blockquotes, inline code, raw HTML, math, table rows.

/\A[#>`<$|]/.freeze
TABLE_SEPARATOR_RE =

GFM table separator rows — "| --- | --- |" or the pipe-only "--- | ---" variant — are dashes, pipes, colons and spaces only. Kept as structure; the noise streams this filter targets always mix in other symbol types (+ = · ~ @ # % …), which this narrow pattern cannot match, so it is safe to exempt the whole class.

/\A\|?[\s\-:|]+\|?\z/.freeze
FENCE_RE =

Fenced code opener/closer: three or more backticks or tildes, optionally with an info string.

/\A(?:`{3,}|~{3,})/.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(min_length: DEFAULT_MIN_LENGTH, max_entropy: DEFAULT_MAX_ENTROPY) ⇒ NoiseFilter

Returns a new instance of NoiseFilter.



65
66
67
68
 
# File 'lib/ask/web_fetch/noise_filter.rb', line 65

def initialize(min_length: DEFAULT_MIN_LENGTH, max_entropy: DEFAULT_MAX_ENTROPY)
  @min_length = min_length
  @max_entropy = max_entropy
end

Class Method Details

.filter(markdown, min_length: DEFAULT_MIN_LENGTH, max_entropy: DEFAULT_MAX_ENTROPY) ⇒ Object

Returns markdown with decorative noise lines removed. The options override the conservative defaults.



60
61
62
 
# File 'lib/ask/web_fetch/noise_filter.rb', line 60

def filter(markdown, min_length: DEFAULT_MIN_LENGTH, max_entropy: DEFAULT_MAX_ENTROPY)
  new(min_length: min_length, max_entropy: max_entropy).filter(markdown)
end

Instance Method Details

#filter(markdown) ⇒ Object 



70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
 
# File 'lib/ask/web_fetch/noise_filter.rb', line 70

def filter(markdown)
  out = +''
  in_fence = false
  in_indented_code = false
  prev_blank = false

  markdown.each_line do |line|
    stripped = line.strip

    # Fenced code: flip on any fence opener/closer, then pass the
    # whole block through untouched — code may legitimately be
    # nothing but symbols.
    if stripped.match?(FENCE_RE)
      in_fence = !in_fence
      out << line
      next
    end
    if in_fence
      out << line
      next
    end

    # Indented code (GFM-ish: 4+ leading spaces, ends at a blank
    # line). Passed through untouched for the same reason.
    if in_indented_code && stripped.empty?
      in_indented_code = false
      out << line
      next
    end
    indented = line.start_with?('    ', "\t")
    if indented && (in_indented_code || prev_blank)
      in_indented_code = true
      out << line
      next
    end

    prev_blank = stripped.empty?
    out << line unless noise_line?(stripped)
  end
  out
end