Module: Pikuri::Extractor::HTML

Defined in:
lib/pikuri/extractor/html.rb

Overview

HTML → Markdown extractor.

Matched by content-type only (+text/html+ / +application/xhtml+xml+) — no byte sniff: the web path always has the header, and sniffing would route a Workspace::Read of a local .html source file through readability when the developer wants the source. Local HTML stays on the Passthrough arm.

Renders both views when available, concatenated with a horizontal rule, so the LLM gets structured metadata and rendered body and picks whichever fits:

  1. JSON-LD — the first <script type="application/ld+json"> node whose @type is a substantive schema.org type (INTERESTING_TYPES), rendered as title + metadata bullets + body copy.
  2. Readability — Readability + reverse_markdown, with a +
    +/+
    + fallback for pages whose content sits outside <p> tags.

Emitting both trades some duplication (a body embedded in JSON-LD and HTML) for dropping type-based "teaser or real body?" heuristics.

Constant Summary collapse

CONTENT_TYPES =

Returns content-types this extractor claims.

Returns:

  • (Array<String>) —

    content-types this extractor claims.

%w[text/html application/xhtml+xml].freeze
INTERESTING_TYPES =

Returns schema.org @type values treated as the page's primary entity when picking a JSON-LD node (first match wins). Skips noise nodes (Organization, BreadcrumbList, WebSite) that carry no content.

Returns:

  • (Array<String>) —

    schema.org @type values treated as the page's primary entity when picking a JSON-LD node (first match wins). Skips noise nodes (Organization, BreadcrumbList, WebSite) that carry no content.

%w[
  Product Article NewsArticle BlogPosting Recipe Event Book Movie
].freeze
READABILITY_TAGS =

Returns HTML tags preserved by the readability pass. Anything outside this list is stripped before Markdown conversion.

Returns:

  • (Array<String>) —

    HTML tags preserved by the readability pass. Anything outside this list is stripped before Markdown conversion.

%w[
  h1 h2 h3 h4 h5 h6 p div span ul ol li blockquote pre code a img
  strong em b i br hr table thead tbody tr td th
].freeze
READABILITY_ATTRS =

Returns HTML attributes preserved by the readability pass; everything else (class, id, style, data-*) is dropped before Markdown conversion.

Returns:

  • (Array<String>) —

    HTML attributes preserved by the readability pass; everything else (class, id, style, data-*) is dropped before Markdown conversion

%w[href src alt title].freeze
MAIN_FALLBACK_RATIO =

Returns minimum +

+/+
+-to-Readability text-length ratio that triggers the semantic-container fallback in readability_to_markdown. Low enough to catch Readability collapsing a div/list-based page (vaadin.com/company, ~5x), high enough to keep its noise filtering where both outputs are comparable.

Returns:

  • (Float) —

    minimum +

    +/+
    +-to-Readability text-length ratio that triggers the semantic-container fallback in readability_to_markdown. Low enough to catch Readability collapsing a div/list-based page (vaadin.com/company, ~5x), high enough to keep its noise filtering where both outputs are comparable.

2.0
MAIN_FALLBACK_MIN_CHARS =

Returns minimum text length the +

+/+
+ container must hold before the readability_to_markdown fallback can fire — below it the ratio is noise-dominated and would swap on tiny pages.

Returns:

  • (Integer) —

    minimum text length the +

    +/+
    + container must hold before the readability_to_markdown fallback can fire — below it the ratio is noise-dominated and would swap on tiny pages.

500

Class Method Summary collapse

Class Method Details

.extract(io) ⇒ String

Render the HTML behind io as Markdown: the JSON-LD section (when an interesting node is present) and the readability/+

+ section, joined by a horizontal rule. Either may be missing.

Parameters:

  • io (IO, StringIO) —

    IO over the HTML document.

Returns:

  • (String) —

    Markdown representation



87
88
89
90
91
92
 
# File 'lib/pikuri/extractor/html.rb', line 87

def self.extract(io)
  html = io.read
  sections = [jsonld_section(html), readability_to_markdown(html)]
  sections.reject! { |s| s.nil? || s.strip.empty? }
  sections.join("\n\n---\n\n")
end

.jsonld_section(html) ⇒ String?

Pick the first JSON-LD node whose @type is in INTERESTING_TYPES and render it. nil when none matches. No content-field gating — a metadata-only node (name/author/date) still renders, since the readability pass independently produces the body.

Parameters:

  • html (String) —

    HTML document body

Returns:

  • (String, nil) —

    Markdown render, or nil when nothing matched



101
102
103
104
105
106
 
# File 'lib/pikuri/extractor/html.rb', line 101

def self.jsonld_section(html)
  node = parse_jsonld(html).find do |n|
    Array(n['@type']).any? { |t| INTERESTING_TYPES.include?(t) }
  end
  node ? jsonld_to_markdown(node) : nil
end

.jsonld_to_markdown(node) ⇒ String

Render a JSON-LD node as Markdown: title from +name+/+headline+, a bullet list of common fields (brand, SKU, price, rating, author, date), body copy, and the lead image. articleBody wins over description (the description is usually a lede teaser).

Parameters:

  • node (Hash) —

    JSON-LD node

Returns:

  • (String) —

    Markdown representation



138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/pikuri/extractor/html.rb', line 138

def self.jsonld_to_markdown(node)
  out = +''
  name = node['name'] || node['headline']
  out << "# #{name}\n\n" if name

  offer  = first_obj(node['offers'])
  rating = first_obj(node['aggregateRating'])
  brand  = first_obj_or_string(node['brand'])
  author = first_obj_or_string(node['author'])

  brand_name  = brand.is_a?(Hash)  ? brand['name']  : brand
  author_name = author.is_a?(Hash) ? author['name'] : author

  fields = {
    'Brand'        => brand_name,
    'SKU'          => node['sku'],
    'GTIN'         => node['gtin13'] || node['gtin'],
    'Price'        => [offer['price'], offer['priceCurrency']].compact.join(' '),
    'Availability' => offer['availability'],
    'Rating'       => rating['ratingValue'],
    'Reviews'      => rating['reviewCount'],
    'Author'       => author_name,
    'Published'    => node['datePublished']
  }.reject { |_, v| v.nil? || v.to_s.strip.empty? }

  unless fields.empty?
    fields.each { |k, v| out << "- **#{k}:** #{v}\n" }
    out << "\n"
  end

  if (body = node['articleBody'] || node['description'])
    out << "#{body}\n\n"
  end

  if (img = node['image'])
    img = img.first if img.is_a?(Array)
    img = img['url'] if img.is_a?(Hash)
    out << "![image](#{img})\n\n" if img
  end

  out
end

.kind ⇒ Symbol

Returns Page#kind tag.

Returns:



69
70
71
 
# File 'lib/pikuri/extractor/html.rb', line 69

def self.kind
  :html
end

.matches?(sample:, content_type:) ⇒ Boolean

Parameters:

  • sample (String) —

    leading bytes of the content (unused — see the no-sniff rationale in the module doc).

  • content_type (String, nil) —

    normalized content-type.

Returns:

  • (Boolean) 


77
78
79
 
# File 'lib/pikuri/extractor/html.rb', line 77

def self.matches?(sample:, content_type:)
  CONTENT_TYPES.include?(content_type)
end

.parse_jsonld(html) ⇒ Array<Hash>

Collect every JSON-LD payload in html, flattening @graph wrappers into one flat array. Malformed JSON blocks are skipped (sites ship broken JSON-LD; one parseable block suffices).

Parameters:

  • html (String) —

    HTML document body

Returns:

  • (Array<Hash>) —

    parsed nodes; possibly empty



114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
 
# File 'lib/pikuri/extractor/html.rb', line 114

def self.parse_jsonld(html)
  doc = Nokogiri::HTML(html)
  blobs = doc.css('script[type="application/ld+json"]').map(&:text)

  blobs.flat_map do |raw|
    parsed = begin
      JSON.parse(raw)
    rescue JSON::ParserError
      nil
    end
    next [] unless parsed

    nodes = parsed.is_a?(Array) ? parsed : [parsed]
    nodes.flat_map { |n| n['@graph'].is_a?(Array) ? n['@graph'] : [n] }
  end
end

.readability_to_markdown(html) ⇒ String

Run Readability over html, convert to Markdown via reverse_markdown, <title> as a top-level heading. When the page uses HTML5 semantics (+

+/+
+) but leaves content outside <p> tags, Readability's paragraph-density scoring collapses the extraction; then the container is rendered directly. The fallback fires only when the container holds substantially more text (see MAIN_FALLBACK_RATIO / MAIN_FALLBACK_MIN_CHARS).

Parameters:

  • html (String) —

    HTML document body

Returns:

  • (String) —

    Markdown representation



191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
 
# File 'lib/pikuri/extractor/html.rb', line 191

def self.readability_to_markdown(html)
  rdoc = Readability::Document.new(
    html,
    tags: READABILITY_TAGS,
    attributes: READABILITY_ATTRS,
    remove_empty_nodes: true
  )
  readability_html = rdoc.content
  title = rdoc.title

  body_html = main_fallback_html(html, readability_html) || readability_html
  body = ReverseMarkdown.convert(body_html, unknown_tags: :bypass, github_flavored: true)

  out = +''
  out << "# #{title.strip}\n\n" if title && !title.strip.empty?
  out << body
  out
end