Class: Docsmith::Diff::Parsers::Markdown

Inherits:
Renderers::Base show all
Defined in:
lib/docsmith/diff/parsers/markdown.rb

Overview

Word-level diff parser for Markdown documents.

Tokenizes into words and newline runs rather than lines, so prose edits are detected at word granularity instead of "the whole line changed".

"Hello world\n\nFoo" → ["Hello", "world", "\n\n", "Foo"]

Grouping, offsets, and rendering all come from Renderers::Base — this class only decides where token boundaries fall. Spaces and tabs between tokens are not themselves tokens, but they are still present in an edit's text, which is sliced from the source between the edit's offsets.

Constant Summary collapse

WORD_OR_NEWLINES =

\S+ matches any non-whitespace run (words, punctuation, markdown markers). \n+ matches consecutive newlines as one token, so a paragraph break (\n\n) and a line break (\n) are each a single diffable unit.

/\S+|\n+/

Constants inherited from Renderers::Base

Renderers::Base::NEW_SIDE, Renderers::Base::OLD_SIDE

Instance Method Summary collapse

Methods inherited from Renderers::Base

#compute, #render_html

Instance Method Details

#tokenize(content) ⇒ Array<Array(String, Integer)>

Returns [token, start_offset] pairs.

Parameters:

  • content (String)

Returns:

  • (Array<Array(String, Integer)>) —

    [token, start_offset] pairs



27
28
29
 
# File 'lib/docsmith/diff/parsers/markdown.rb', line 27

def tokenize(content)
  scan_with_offsets(content, WORD_OR_NEWLINES)
end