Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
"UTF-8" support was often just single byte, so it was little more than ASCII
"Single byte UTF-8" is ASCII. That's one of its most important properties.
Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing
How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).
Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.
Allowing more than one lead byte is a complication of the existing standard. As many others have pointed out, we have plenty of coding space without that (e.g. by allowing 5- and 6-byte UTF-8 again), so the case for the extra complexity is currently not compelling.
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
The number of glyphs available by adding additional bytes drops exponentially because each subsequent byte has one less bit available.
So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.
I don't quite get it. 5-byte utf-8 encoding gets extra 5 bits compared to 4 byte, and 6-byte gets extra 10 bits. If you were thinking about bits in leading byte, then yes, you are losing one bit for every extra trailing byte, but you also get 6 bits from it. So adding a byte gives you extra 5 bits.
On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
"your first 2 extensions (5 and 6 bytes) were clearly envisioned.
the standard (up to 4 bytes) was created to cover the size of
unicode. i thought any more description would be a waste of
paper. i think your extension from 7 to 8 bytes is a little hoaky.
i requires reading the whole string rather than "knowing" the
number of follow on bytes. so, i think the only thing new is the
7 byte version.
i appreciate the mail, but i really dont think it is useful. it is
like replacing ipv6 with ipv50."
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.
Isn't a limit by definition no longer dependent on n?
U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.
2-3 bytes are not much space for anything. Sure, you could use multiple successive ones of these code points and define your own "continuation" encoding in these ranges, but that doesn't seem right to me somehow
I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course allowing a script to be written in a single codepoint is the much more elegant solution.
It also neatly solves the problem of how to write Unicode strings inside scripts inside Unicode strings and also scripts inside Unicode strings inside scripts inside Unicode strings.
Another one: encode instructions on how to draw the glyph into the text itself. The string becomes both the text and the font. Why not make it turing complete and as powerful/complex as TTF.
Imagine someone using the same fully customized emoji multiple times in the same text. Seems like a waste of space. Maybe better to encode just a UUID, and send the image codebook separately.
Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.
I don’t see how UTF-8000 doesn’t have it. The first byte of any code point is either 0xxxxxxx or 10xxxxxxx, which is distinct from all non-first bytes which are 11xxxxxx. Thus any UTF-8000 sub-bytestring must necessarily have the start aligned at a code point boundary, at which point all the subsequent bytes are interpreted as codepoints in the same way.
I was about to immediately suggest UCS-X, and then pleased that the author offers a fair comparison to UCS-X and many other alternatives. Great. Too bad UCS-X is possibly more memorable name.
IIRC UTF-8 proposal used to go to 6 bytes as they already felt that not having part of the code point in the first byte was an issue somehow.
I think at least 7 should be allowed. 42 bits could contain all sorts of information.
"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
For compatibility with UTF-16:
* https://datatracker.ietf.org/doc/html/rfc3629#section-12
* https://en.wikipedia.org/wiki/UTF-16
The original spec had 31 bits (the UTF-32/UCS-4 range):
* https://datatracker.ietf.org/doc/html/rfc2279
* https://en.wikipedia.org/wiki/UTF-32
FF bytes are an easy way to identify an invalid UTF-8 file. This idea doesn't have that property.
True, but not all non-UTF8 bytestrings contain 0xFF bytes, so it’s not very useful in practice.
Yes, I agree.
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
"Single byte UTF-8" is ASCII. That's one of its most important properties.
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).
Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
As are FE, FD, FC, FB, FA, F9, F8, F7, F6 and F5.
No project is ever safe from complicators.
This is why we need the KISS enforcers.
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.
Allowing more than one lead byte is a complication of the existing standard. As many others have pointed out, we have plenty of coding space without that (e.g. by allowing 5- and 6-byte UTF-8 again), so the case for the extra complexity is currently not compelling.
Phew, and I was worried that we'd be running out of UNICODE space for new emojis ;)
I love it.
Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.
More like WTF-8.
https://xkcd.com/1953/
No, WTF-8 is its own encoding: https://wtf-8.codeberg.page/
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
We still have about 85% of codepoint space unused. Hopefully, by the time it becomes a problem, UTF-16 will be long dead
i hope so too, but UTF-16 being used by languages such as java and javascript makes me fear it might be here to stay.... i hope im wrong
The number of glyphs available by adding additional bytes drops exponentially because each subsequent byte has one less bit available.
So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.
I don't quite get it. 5-byte utf-8 encoding gets extra 5 bits compared to 4 byte, and 6-byte gets extra 10 bits. If you were thinking about bits in leading byte, then yes, you are losing one bit for every extra trailing byte, but you also get 6 bits from it. So adding a byte gives you extra 5 bits.
Yeah, you’re right. I might have attempted to do mental arithmetic before coffee…
On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
The letter from Ken Thompson is spot on:
"your first 2 extensions (5 and 6 bytes) were clearly envisioned. the standard (up to 4 bytes) was created to cover the size of unicode. i thought any more description would be a waste of paper. i think your extension from 7 to 8 bytes is a little hoaky. i requires reading the whole string rather than "knowing" the number of follow on bytes. so, i think the only thing new is the 7 byte version.
i appreciate the mail, but i really dont think it is useful. it is like replacing ipv6 with ipv50."
Just limit it to 8 bytes at which point you always do 'know the number of follow on bytes' from the first byte.
Nobody needs more than 4.47 trillion characters. (famous last words)
This is like TIME CUBE (https://en.wikipedia.org/wiki/Time_Cube) but for encoding nerds.
edit: replaced false prophet site with wiki link for the original one and true timecube.com...lost to the sands of ...time.
timecube.com is lost, zombo.com has been redesigned. If this is not proof of a dead internet, I don’t know what is.
It would be nice if the first paragraph gave a motivation for this. Why did you start this?
I mean, I can derive from the text that this can encode arbitrarily large integers. So what? What's the point? I also understand that this is for fun.
Yes, sure. But what was the incentive? The goal? The reason to start the project (and not another one)? What was interesting enough to start this?
Some ideas of what to do with this space:
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
Isn't a limit by definition no longer dependent on n?
U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.
The point is probably to encode the emoji (image) in the codepoint itself.
2-3 bytes are not much space for anything. Sure, you could use multiple successive ones of these code points and define your own "continuation" encoding in these ranges, but that doesn't seem right to me somehow
I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course allowing a script to be written in a single codepoint is the much more elegant solution.
It also neatly solves the problem of how to write Unicode strings inside scripts inside Unicode strings and also scripts inside Unicode strings inside scripts inside Unicode strings.
(*) in RFC 04-01-2027
[1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)
One more idea:
- Unicode-as-a-character, meaning: a whole text displayed as if it were one character, made for infinite zooming
Another one: encode instructions on how to draw the glyph into the text itself. The string becomes both the text and the font. Why not make it turing complete and as powerful/complex as TTF.
Imagine someone using the same fully customized emoji multiple times in the same text. Seems like a waste of space. Maybe better to encode just a UUID, and send the image codebook separately.
Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
Only if they've already both undergone normalization to NFC or NFD.
Rely on it? Not that I can remember.
However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.
I don’t see how UTF-8000 doesn’t have it. The first byte of any code point is either 0xxxxxxx or 10xxxxxxx, which is distinct from all non-first bytes which are 11xxxxxx. Thus any UTF-8000 sub-bytestring must necessarily have the start aligned at a code point boundary, at which point all the subsequent bytes are interpreted as codepoints in the same way.
I was about to immediately suggest UCS-X, and then pleased that the author offers a fair comparison to UCS-X and many other alternatives. Great. Too bad UCS-X is possibly more memorable name.
Finally, an ASCII-safe way to encode all the vectors of Neuralese.
IIRC UTF-8 proposal used to go to 6 bytes as they already felt that not having part of the code point in the first byte was an issue somehow.
I think at least 7 should be allowed. 42 bits could contain all sorts of information.
"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.
Alright, so now we can fit an entire LLM model into one character.
I don’t actually know if this is LLM-generated, but phrasing like this is weirdly triggering to me now
Yeah that kind of line is what I see all the time in my chats. Even worse worse is when they put it in code comments.