🤨 WTF-8 for Ruby
A superset of UTF-8 that encodes surrogate code points if they are not in a pair.
What is WTF-8 for Ruby?
Windows filenames, JavaScript strings, Java strings, and JSON \uD800 escapes are all sequences of 16-bit code units that are only potentially UTF-16. An unpaired surrogate is a legal value in all of them, but it isn't a Unicode scalar value, so UTF-8 can't encode it. Ruby refuses to convert one:
[0xD800].pack("v").force_encoding("UTF-16LE").encode("UTF-8")
# Encoding::InvalidByteSequenceError: incomplete "\x00\xD8" on UTF-16LE
WTF-8 fills that gap. It encodes a surrogate the same way UTF-8 encodes any other code point, and encodes a surrogate pair as the single four-byte sequence for the code point it represents. That second rule is what makes conversion to and from potentially ill-formed UTF-16 lossless in both directions.
This gem implements the WTF-8 specification by Simon Sapin. Rust uses WTF-8 internally to represent Windows paths, and there is a Rust implementation by the same author.
WTF8.from_utf16le([0xD800].pack("v")).code_points
#=> [55296]
Installation
bundle add wtf8
Pure Ruby, no dependencies and no native extension. Requires Ruby 3.2 or later.
Usage
Reading UTF-16
WTF8.from_utf16le(bytes) # also from_utf16be and from_utf16
WTF8.from_utf16([0xD83D, 0xDE00]) #=> #<WTF8::String "😀">
WTF8.from_utf16([0xD83D]) #=> #<WTF8::String "\u{D83D}">
UTF-8 is a subset of WTF-8, so reading a UTF-8 string costs nothing:
WTF8.from_utf8("héllo 😀")
Converting back
to_utf16 is exact. to_utf8 isn't, since UTF-8 has no encoding for a surrogate, so each one is replaced with U+FFFD.
string = WTF8.from_code_points([0x61, 0xD800, 0x62])
string.to_utf16 #=> [97, 55296, 98]
string.to_utf8 #=> "a�b"
string.to_utf8(replacement: "?") #=> "a?b"
One surrogate costs one replacement character. Ruby's String#scrub sees three invalid bytes rather than one invalid code point, and spends three:
"\xED\xA0\x80".dup.force_encoding("UTF-8").scrub("?") #=> "???"
WTF8::String
A Ruby String can store these bytes, since it's bytes plus an encoding tag, but it can't interpret them. Tagged as UTF-8, three bytes of lone surrogate have a length of 3 and raise from codepoints.
string = WTF8.from_code_points([0x61, 0xD83D])
string.length #=> 2
string.code_points #=> [97, 55357]
string.well_formed? #=> false
string.raw.dup.force_encoding("UTF-8").length #=> 4
Instances are frozen, validated on construction, and compare and hash by their bytes.
Concatenation
Concatenating WTF-8 isn't the same as concatenating bytes. If the left side ends with a lead surrogate and the right side starts with a trail surrogate, the two form a pair, and a pair has to be encoded as one four-byte sequence. Six bytes become four, and the result is one code point shorter than its halves were:
left = WTF8.from_code_points([0x61, 0xD83D])
right = WTF8.from_code_points([0xDE00, 0x62])
left.length + right.length #=> 4
(left + right).length #=> 3
(left + right).to_s #=> "a😀b"
Slicing is the same thing in reverse. It's defined in UTF-16 code units, because that's the only index space where the halves of a supplementary code point can be addressed:
emoji = WTF8.from_utf8("a😀b")
emoji.slice_utf16(0..1) #=> #<WTF8::String "a\u{D83D}">
emoji.slice_utf16(2..3) #=> #<WTF8::String "\u{DE00}b">
emoji.slice_utf16(0..1) + emoji.slice_utf16(2..3) == emoji #=> true
CESU-8 and Modified UTF-8
CESU-8 inverts the rule WTF-8 uses for supplementary code points. It always encodes them as two surrogates of three bytes each, where WTF-8 always encodes them as one four-byte sequence. Java's Modified UTF-8 is CESU-8 with U+0000 encoded as C0 80, so that an encoded string never contains a NUL byte. DataOutputStream.writeUTF, .class constant pools, dex files and JNI all use it.
WTF8.from_cesu8(bytes)
WTF8.from_modified_utf8(bytes)
WTF8.from_utf8("😀").to_cesu8.bytes #=> [237, 160, 189, 237, 184, 128]
Ruby has a built-in CESU-8 transcoder, and for well-formed input this produces identical bytes. Ruby's rejects lone surrogates in both directions; this one doesn't.
Working with bytes
WTF8::Codec offers the same operations on plain binary strings, without the wrapper object:
WTF8::Codec.decode("\xED\xA0\x80") #=> [55296]
WTF8::Codec.encode([0xD800]).bytes #=> [237, 160, 128]
WTF8::Codec.concat(left, right)
WTF8::Codec.valid?("\xED\xA0\xBD\xED\xB8\x80") #=> false
Don't use it for interchange
WTF-8 is an internal representation. Reading UTF-8 as WTF-8 is safe, but WTF-8 is not UTF-8 and must not be sent as though it were. The byte sequences it adds are exactly the ones a UTF-8 decoder is required to reject, and inconsistent handling of them across a pipeline is a known validation-bypass vector.
Convert with to_utf8 at the boundary. Don't serve it as charset=utf-8, don't write it to a file another program will read as UTF-8, and don't store it in a UTF-8 column.
License
The gem is available as open source under the terms of the MIT License.