Module: HtmlToPlainText

Defined in:
lib/html_to_plain_text.rb

Overview

The main method on this module plain_text will convert a string of HTML to a plain text approximation.

Constant Summary collapse

IGNORE_TAGS = 
%w[script noscript style object applet iframe template svg math canvas audio video select option optgroup datalist textarea].each_with_object({}) { |t, h|
  h[t] = true
}.freeze
PARAGRAPH_TAGS = 
%w[p h1 h2 h3 h4 h5 h6 table ol ul menu dl dd blockquote dialog figure aside section].each_with_object({}) { |t, h|
  h[t] = true
}.freeze
BLOCK_TAGS = 
%w[div address li dt center del article header footer nav pre legend tr main figcaption caption summary details form fieldset hgroup].each_with_object({}) { |t, h|
  h[t] = true
}.freeze
%w[header footer nav].each_with_object({}) { |t, h|
  h[t] = true
}.freeze
%w[navigation banner contentinfo].each_with_object({}) { |t, h|
  h[t] = true
}.freeze
MARKDOWN_INLINE_TAGS = 
{
  "strong" => "**",
  "b" => "**",
  "em" => "*",
  "i" => "*",
  "del" => "~~",
  "s" => "~~",
  "strike" => "~~",
  "code" => "`"
}.freeze
MARKDOWN_HEADING_TAGS = 
{
  "h1" => "# ",
  "h2" => "## ",
  "h3" => "### ",
  "h4" => "#### ",
  "h5" => "##### ",
  "h6" => "###### "
}.freeze
WHITESPACE = 
[" ", "\n", "\r"].freeze
PLAINTEXT = 
"plaintext"
PRE = 
"pre"
BR = 
"br"
HR = 
"hr"
TD = 
"td"
TH = 
"th"
TR = 
"tr"
OL = 
"ol"
UL = 
"ul"
LI = 
"li"
A = 
"a"
IMG = 
"img"
"menu"
BLOCKQUOTE = 
"blockquote"
TABLE = 
"table"
THEAD = 
"thead"
TBODY = 
"tbody"
NUMBERS = 
["1", "a"].freeze
/\A(?:[a-z][a-z0-9.+-]*:\/\/[a-z0-9]|mailto:|tel:)/i
URI_SCHEME_PATTERN = 
/\A[a-z][a-z0-9.+-]*:/i
BRACKET_PATTERN = 
/[\[\]]/
BACKTICK_RUN_PATTERN = 
/`+/
HTML_PATTERN = 
/[<&]/
BODY_TAG_XPATH = 
"/html/body"
FIRST_TR_XPATH = 
".//tr"
CARRIAGE_RETURN_PATTERN = 
/\r\n?/
LINE_BREAK_PATTERN = 
/[\n\r]/
NON_PROTOCOL_PATTERN = 
/:\/?\/?(.*)/
ALL_WHITESPACE_PATTERN = 
/[[:space:]]+/
NOT_WHITESPACE_PATTERN = 
/[^[:space:]]/
LEADING_WHITESPACE_PATTERN = 
/\A[[:space:]]*/
TRAILING_WHITESPACE_PATTERN = 
/[[:space:]]*\z/
LEADING_LINE_BREAK_PATTERN = 
/\A[\r\n]+/
SPACE = 
" "
TAB = 
"\t"
EMPTY = 
""
NEWLINE = 
"\n"
PIPE = 
"|"
ESCAPED_PIPE = 
"\\|"
HREF = 
"href"
SRC = 
"src"
ALT = 
"alt"
BORDER = 
"border"
ROLE = 
"role"
TABLE_SEPARATOR = 
" | "
MARKDOWN_HR = 
"---\n"
MARKDOWN_BR_MARKER =

Hard line breaks are emitted as a marker character so that ones that would produce a stray backslash (before a blank line or at the end of the output) can be removed when the output is finalized. The null character cannot appear in parsed HTML text.

"\u0000"
MARKDOWN_BR = 
"#{MARKDOWN_BR_MARKER}\n"
MARKDOWN_HARD_BREAK = 
"\\"
MARKDOWN_BR_BEFORE_BLANK_PATTERN = 
/#{MARKDOWN_BR_MARKER}(?=\n[>[[:space:]]]*\n)/
MARKDOWN_FENCE = 
"```"
MARKDOWN_QUOTE = 
"> "
MARKDOWN_EMPTY_QUOTE = 
">"
MARKDOWN_BULLET = 
"- "
BACKTICK = 
"`"
MARKDOWN_TABLE_SEPARATOR_CELL = 
" --- |"

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.markdown(html, show_links: true, all_tables: false, ignore_nav: false, selector: nil) ⇒ String

Convert some HTML into a Markdown approximation. This is the same as calling plain_text with the markdown option set to true.

Parameters:

  • html (String) —

    The HTML to convert into Markdown.

  • show_links (Boolean) (defaults to: true) —

    Whether to include link URLs and image sources in the output.

  • all_tables (Boolean) (defaults to: false) —

    Whether to format all tables as data tables regardless of their markup.

  • ignore_nav (Boolean) (defaults to: false) —

    Whether to suppress navigational, header, and footer elements.

  • selector (String, nil) (defaults to: nil) —

    A CSS selector limiting the output to matching elements.

Returns:

  • (String) —

    The Markdown approximation of the HTML.

Raises:

  • (ArgumentError) —

    If the selector is not a valid CSS selector.



195
196
197
 
# File 'lib/html_to_plain_text.rb', line 195

def markdown(html, show_links: true, all_tables: false, ignore_nav: false, selector: nil)
  plain_text(html, show_links: show_links, markdown: true, all_tables: all_tables, ignore_nav: ignore_nav, selector: selector)
end

.plain_text(html, show_links: true, markdown: false, all_tables: false, ignore_nav: false, selector: nil) ⇒ String

Convert some HTML into a plain text approximation.

Parameters:

  • html (String) —

    The HTML to convert into plain text.

  • show_links (Boolean) (defaults to: true) —

    Whether to include link URLs and image sources in the output.

  • markdown (Boolean) (defaults to: false) —

    Whether to format the output as Markdown.

  • all_tables (Boolean) (defaults to: false) —

    Whether to format all tables as data tables regardless of their markup. By default only tables with a non-zero border attribute or a thead or tbody element are formatted as data tables; other tables are assumed to be for layout only.

  • ignore_nav (Boolean) (defaults to: false) —

    Whether to suppress navigational, header, and footer elements. When true, header, footer, and nav tags are omitted from the output along with any elements that have a role attribute of navigation, banner, or contentinfo.

  • selector (String, nil) (defaults to: nil) —

    A CSS selector limiting the output to matching elements. Only the contents of elements matching the selector are included in the output. Only elements within the body of the document are matched.

Returns:

  • (String) —

    The plain text approximation of the HTML.

Raises:

  • (ArgumentError) —

    If the selector is not a valid CSS selector.



145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
 
# File 'lib/html_to_plain_text.rb', line 145

def plain_text(html, show_links: true, markdown: false, all_tables: false, ignore_nav: false, selector: nil)
  return nil if html.nil?
  unless selector || HTML_PATTERN.match?(html)
    return html.gsub(CARRIAGE_RETURN_PATTERN, NEWLINE).strip
  end
  document = Nokogiri::HTML::Document.parse(html)
  options = {show_links: show_links, markdown: markdown, all_tables: all_tables, ignore_nav: ignore_nav}
  out = +""
  body = document.xpath(BODY_TAG_XPATH).first
  return +"" unless body
  if selector
    begin
      elements = document.css(selector)
    rescue Nokogiri::CSS::SyntaxError => e
      raise ArgumentError, "Invalid CSS selector: #{e.message}"
    end
    elements.each do |element|
      ancestors = element.ancestors
      # Only elements within the body are included, and nested matches are
      # skipped since their content is already included by a matching ancestor.
      next unless element == body || ancestors.include?(body)
      next if ancestors.any? { |ancestor| elements.include?(ancestor) }
      append_block_breaks(out)
      convert_node_to_plain_text(element, out, options)
    end
  else
    convert_node_to_plain_text(body, out, options)
  end
  # String#strip removes null characters as well as whitespace, so a hard break
  # marker at the end of the output is removed here along with any trailing newline.
  out = out.strip
  if markdown
    # Hard break markers followed by a blank line would leave a stray backslash,
    # so they are removed; the rest become hard line break backslashes.
    out.gsub!(MARKDOWN_BR_BEFORE_BLANK_PATTERN, EMPTY)
    out.gsub!(MARKDOWN_BR_MARKER, MARKDOWN_HARD_BREAK)
  end
  out.gsub(CARRIAGE_RETURN_PATTERN, NEWLINE)
end

Instance Method Details

#markdown(html, show_links: true, all_tables: false, ignore_nav: false, selector: nil) ⇒ String

Helper instance method for converting HTML into Markdown. This method simply calls HtmlToPlainText.markdown.

Parameters:

  • html (String) —

    The HTML to convert into Markdown.

  • show_links (Boolean) (defaults to: true) —

    Whether to include link URLs and image sources in the output.

  • all_tables (Boolean) (defaults to: false) —

    Whether to format all tables as data tables regardless of their markup.

  • ignore_nav (Boolean) (defaults to: false) —

    Whether to suppress navigational, header, and footer elements.

  • selector (String, nil) (defaults to: nil) —

    A CSS selector limiting the output to matching elements.

Returns:

  • (String) —

    The Markdown approximation of the HTML.

Raises:

  • (ArgumentError) —

    If the selector is not a valid CSS selector.



124
125
126
 
# File 'lib/html_to_plain_text.rb', line 124

def markdown(html, show_links: true, all_tables: false, ignore_nav: false, selector: nil)
  HtmlToPlainText.markdown(html, show_links: show_links, all_tables: all_tables, ignore_nav: ignore_nav, selector: selector)
end

#plain_text(html, show_links: true, markdown: false, all_tables: false, ignore_nav: false, selector: nil) ⇒ String

Helper instance method for converting HTML into plain text. This method simply calls HtmlToPlainText.plain_text.

Parameters:

  • html (String) —

    The HTML to convert into plain text.

  • show_links (Boolean) (defaults to: true) —

    Whether to include link URLs and image sources in the output.

  • markdown (Boolean) (defaults to: false) —

    Whether to format the output as Markdown.

  • all_tables (Boolean) (defaults to: false) —

    Whether to format all tables as data tables regardless of their markup.

  • ignore_nav (Boolean) (defaults to: false) —

    Whether to suppress navigational, header, and footer elements.

  • selector (String, nil) (defaults to: nil) —

    A CSS selector limiting the output to matching elements.

Returns:

  • (String) —

    The plain text approximation of the HTML.

Raises:

  • (ArgumentError) —

    If the selector is not a valid CSS selector.



111
112
113
 
# File 'lib/html_to_plain_text.rb', line 111

def plain_text(html, show_links: true, markdown: false, all_tables: false, ignore_nav: false, selector: nil)
  HtmlToPlainText.plain_text(html, show_links: show_links, markdown: markdown, all_tables: all_tables, ignore_nav: ignore_nav, selector: selector)
end