Class: Rouge::Lexers::Carve

Inherits:
RegexLexer
  • Object
show all
Defined in:
lib/rouge/lexers/carve.rb

Overview

Carve is a post-Markdown lightweight markup language. Its inline delimiters deliberately differ from Markdown's, which is why lexing a Carve document as Markdown produces actively WRONG output rather than merely plain text:

Carve        means            Markdown would read it as
---------    -------------    -------------------------
*bold*       strong           emphasis
/italic/     emphasis         literal slashes
_under_      underline        emphasis
~strike~     strikethrough    subscript / literal
{=mark=}     highlight        literal braces
{^sup^}      superscript      literal braces

WHY THE BLOCK OPENERS ARE NOT ANCHORED AT COLUMN 0. Carve opens a block at column 0, or at an enclosing container's content column - nowhere in between. So # H at document level is a paragraph, while the same opener at a list item's content column is a real heading. Telling those apart needs a container model that tracks the item's content column. A regex lexer's state stack could carry one, but the sibling grammars (Prism, highlight.js, Pygments) do not, and this lexer keeps their trade-off on purpose so they agree: block openers match at any indent and knowingly over-colour the rare indented-at-document-level case, rather than under-colouring the common valid shape of an indented construct inside a list item.

WHY AN ATTRIBUTE BLOCK IS ONE TOKEN. {#id .cls key="v" :lang} is emitted whole as Name::Attribute rather than split into id, class, key, value and language parts. Splitting reads better in isolation, but an attribute block can carry a brace inside a quoted value, a language tag, and a bare key that is not an attribute at all, and the sibling grammars treat the block as one unit; a consumer asking "is this text inside an attribute block" must get the same answer here as it does there.

WHERE ROUGE CANNOT SAY WHAT CARVE MEANS. Carve's _x_ is UNDERLINE, not emphasis, and Rouge has no underline token - the vocabulary stops at Generic::Emph, Generic::Strong and Generic::EmphStrong. The content takes Generic::Emph so it is at least marked up, and the delimiters stay Punctuation so a consumer can still tell underline from italic by the delimiter it carries. Pygments, which has Generic.Underline, does make the distinction.

Spec: https://markup-carve.github.io/carve/

Constant Summary collapse

MARGIN =

A leading margin. A byte order mark at the start of a document is not content, so a block opener behind one is still a block opener.

/[ \t]*/.freeze
ATTRS =

One attribute block, brace to brace. Quoted values may contain a brace and an escaped quote, so the value alternatives come before the bare-character one; a single nested brace pair is allowed for a braced span written inside.

%r/
  \{(?=[.:}'"]|\#[\w-]+[\s}]|[A-Za-z][\w-]*(?:[=\s}]|$))(?:
    "(?:[^"\\\n]|\\.)*"
    |'(?:[^'\\\n]|\\.)*'
    |\{[^{}\n]*\}
    |[^{}\n]
  )*\}
/x.freeze
ATTRS_LINE =

The same block as a STANDALONE ATTRIBUTE LINE, which may span lines. The inline form deliberately cannot, because an unclosed inline { would otherwise swallow the rest of the document.

%r/
  \{(?=[.:}'"]|\#[\w-]+[\s}]|[A-Za-z][\w-]*(?:[=\s}]|$))(?:
    "(?:[^"\\]|\\.)*"
    |'(?:[^'\\]|\\.)*'
    |\{[^{}]*\}
    |[^{}]
  )*\}
/x.freeze
LABEL =

A bracketed label that may itself contain three levels of brackets. A regex cannot match arbitrarily balanced brackets, and a link label in practice nests a level or two ([t[z]](/u)). Bounding the nesting keeps the common shapes matching instead of stopping at the first inner bracket, which is what a naive [[^]]*] does.

begin
  inner = '[^\[\]\\\\\n]|\\\\.'
  pattern = "(?:#{inner})*"
  3.times { pattern = "(?:#{inner}|\\[#{pattern}\\])*" }
  Regexp.new("\\[#{pattern}\\]").freeze
end
INLINE_STARTERS =

Characters that can begin an inline construct. A run of anything else is ordinary content and is emitted as ONE token - without this every content character becomes its own token, which is both noisy and unusable to a consumer asking whether a phrase carries a scope.

'\\\\%!`${\\[\\^<:@\\#*\\/_~=.\\-'
PLAIN = 
plain_run.freeze
PLAIN_NO_PIPE = 
plain_run('|').freeze
PLAIN_NO_BRACKET = 
plain_run('\\]').freeze

Class Method Summary collapse

Class Method Details

.plain_run(extra = '') ⇒ Object

A run of content characters, excluding extra as well.



106
107
108
 
# File 'lib/rouge/lexers/carve.rb', line 106

def self.plain_run(extra = '')
  Regexp.new("[^\n#{INLINE_STARTERS}#{extra}]+")
end