Module: Marcel::Magic::Xml

Defined in:

lib/marcel/magic/xml.rb

Overview

Procedural refinement of XML documents by their root element.

Many XML vocabularies — feeds, KML, property lists, XSLT, Office 2003 XML — share the same leading bytes, and a prolog of arbitrary length (declaration, comments, DOCTYPE) can precede the element that tells them apart. Apache Tika identifies them by the root element's namespace and local name instead, and those rules are generated into ROOT_XML. This module scans a bounded prefix past the prolog to the root start-tag, resolves the root's namespace from its own xmlns declarations, and looks the pair up to refine a generic application/xml match.

Tika hands the prefix to a namespace-aware SAX parser and takes no root when parsing fails before the first start element, so the scan holds the same line at the token level. Input is brought to UTF-8 before scanning — UTF-16 strictly, legacy encodings with invalid sequences replaced and the document refused if a replacement lands before the end of the root start-tag — so the bytes consumed to that point must be valid in the document's encoding and free of characters the XML version forbids. The XML declaration may appear only first, parsed in full and limited to versions 1.0 and 1.1; no other processing instruction may use the reserved xml target, and a PI's data must be separated from its target by whitespace; comments may not contain "--". The root and attribute QNames and namespace prefixes are validated as Unicode NCNames, while PI targets and reference names admit the full XML Name, whose colon namespace processing leaves alone. Attribute values may not contain "<"; their character references must denote characters the XML version admits, and their entity references must be predefined unless a DTD that could declare them was seen. References are validated, never resolved; nothing beyond the tokens is — no entities, no DTD declarations, no external resources — and anything the scanner cannot read with certainty leaves the generic type in place.

Constant Summary

REFINABLE_TYPE =

"application/xml"

MAX_SCAN =

Tika examines at most this much of a document for its root element.

64 * 1024

UTF8_BOM =

"\xEF\xBB\xBF".b

UTF16LE_BOM =

"\xFF\xFE".b

UTF16BE_BOM =

"\xFE\xFF".b

NAME =

Byte-level scanner pattern for names: XML Name restricted to ASCII plus any non-ASCII byte. It only locates tokens; every captured name is then validated against the real Unicode name grammar (NCNAME and XML_NAME below) after decoding.

'[A-Za-z_:\x80-\xFF][A-Za-z0-9._:\-\x80-\xFF]*'

SPACE =

'[ \t\r\n]'

ENCODING_NAME =

'[A-Za-z][A-Za-z0-9._\-]*'

WHITESPACE =

/\G#{SPACE}*/n

START_TAG =

/\G<(#{NAME})/n

ATTRIBUTE =

/\G#{SPACE}+(#{NAME})#{SPACE}*=#{SPACE}*(?:"([^"<]*)"|'([^'<]*)')/n

START_TAG_END =

/\G#{SPACE}*\/?>/n

PROCESSING_INSTRUCTION =

/\G<\?(#{NAME})/n

RESERVED_TARGET =

/\A[Xx][Mm][Ll]\z/n

XML_DECLARATION =

The complete declaration grammar: a SAX-supported version, then optionally encoding, then optionally standalone, in that order, with nothing else before ?>. Matched in full so that trailing or duplicated tokens cannot hide behind the PI skipper.

/
  \A<\?xml
  #{SPACE}+version#{SPACE}*=#{SPACE}*(?:"(?<version>1\.[01])"|'(?<version>1\.[01])')
  (?:#{SPACE}+encoding#{SPACE}*=#{SPACE}*(?:"(?<encoding>#{ENCODING_NAME})"|'(?<encoding>#{ENCODING_NAME})'))?
  (?:#{SPACE}+standalone#{SPACE}*=#{SPACE}*(?:"(?:yes|no)"|'(?:yes|no)'))?
  #{SPACE}*\?>
/xn

REFERENCE =

Entity and character references; anything else after & in an attribute value is a well-formedness error. The CharRef grammar puts no cap on digits — leading zeros are legal in any number — so digit runs are unbounded here and their magnitude is bounded lexically by valid_character_reference?.

/&(?:(?<name>#{NAME})|\#(?<decimal>[0-9]+)|\#x(?<hex>[0-9A-Fa-f]+));/n

PREDEFINED_ENTITIES =

The five entities every XML document predefines; any other entity reference is only potentially declared when the document carries a DTD.

%w( amp lt gt apos quot ).freeze

NAME_START_CHARS =

NameStartChar and NameChar under the XML 1.0 (5th ed.) Unicode name grammar, colon excluded, matched against decoded UTF-8 names.

"A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD\u{10000}-\u{EFFFF}"

NAME_CHARS =

"#{NAME_START_CHARS}\\-.0-9\u00B7\u0300-\u036F\u203F-\u2040"

NCNAME =

QName parts and namespace prefixes must be NCNames — namespace processing claims their colon — where PI targets and entity names keep the full Name grammar, in which the colon is an ordinary name character.

/\A[#{NAME_START_CHARS}][#{NAME_CHARS}]*\z/

XML_NAME =

/\A[:#{NAME_START_CHARS}][:#{NAME_CHARS}]*\z/

FORBIDDEN_CHARS =

Characters each XML version forbids as literals, matched against the decoded UTF-8 text. XML 1.1 restricts the C0 and C1 controls (NEL excepted) to character references, where XML 1.0 forbids C0 outright but admits C1 literals.

{
  "1.0" => /[\x00-\x08\x0B\x0C\x0E-\x1F\uFFFE\uFFFF]/,
  "1.1" => /[\x00-\x08\x0B\x0C\x0E-\x1F\u007F-\u0084\u0086-\u009F\uFFFE\uFFFF]/,
}.freeze

XML_NAMESPACE =

"http://www.w3.org/XML/1998/namespace"

XMLNS_NAMESPACE =

"http://www.w3.org/2000/xmlns/"

PROCESSING_INSTRUCTION_OPEN =

"<?".b

PROCESSING_INSTRUCTION_CLOSE =

"?>".b

COMMENT_OPEN =

"<!--".b

COMMENT_CLOSE =

"-->".b

DOUBLE_HYPHEN =

"--".b

DOCTYPE_OPEN =

"<!DOCTYPE".b

DOCTYPE_DELIMITER =

/["'\[\]<>]/n

SPACE_BYTES =

[0x20, 0x09, 0x0D, 0x0A].freeze

PSEUDO_ENCODING_NAMES =

Ruby resolves these to a process default rather than to a charset.

%w( locale external filesystem ).freeze

Class Method Summary

• .refine(io, base_type) ⇒ Object

  Returns the type ROOT_XML assigns to the IO's root element if base_type is the generic XML type and the root element can be read, or base_type unchanged.

Class Method Details

.refine(io, base_type) ⇒ Object

Returns the type ROOT_XML assigns to the IO's root element if base_type is the generic XML type and the root element can be read, or base_type unchanged. Partial reads, roots past MAX_SCAN, malformed or mis-encoded prologs, namespace errors and unknown roots refine nothing: the base type stands.

# File 'lib/marcel/magic/xml.rb', line 116

def refine(io, base_type)
  return base_type unless base_type == REFINABLE_TYPE

  io = StringIO.new(io.to_s) unless io.respond_to?(:read)

  root = begin
    if decoded = decode(read_prefix(io))
      root_element(*decoded)
    end
  rescue StandardError
    nil
  ensure
    begin
      io.rewind
    rescue StandardError
      nil
    end
  end

  type = root && ROOT_XML[root]
  type ? Magic.canonical(type) : base_type
end