Yeah this site has a severe issue of some kind that I don't think I've ever seen before. NGL, I'm pretty impressed! It takes so much compute that scrolling lags, which feels very, very strange
Completely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".
Perhaps relevant: Chinese language is not more efficient than English in vibe coding: A preliminary study on token cost and problem-solving ratehttps://arxiv.org/abs/2604.14210v1
This is interesting, thank you. It also reinforces the stocastic parrot theory of LLM. Maybe one mythical day when the majority of codebases around the world are written in CJK Han, vibe coding will become more efficient in Chinese
How does it say anything about the stochastic parrot theory of LLMs?
The only thing there that seems like it's relevant is the finding that the models they used had worse results when prompted in Chinese than when prompted in English. But ...
I am fairly sure I would do substantially worse work if I had to do it in French rather than in English, even if the work itself was all mathematics and programming and the like. Maybe that in some sense indicates that I am a stochastic parrot but it clearly doesn't indicate that I'm a stochastic parrot in some way worse than human beings are since I happen to be a human being myself.
... so what am I missing, that makes the models' worse performance in Chinese an indication that they're stochastic parrots in any interesting sense?
(I take it that "they're stochastic parrots just like we are" is not a very interesting sense. I mean, actually it would be quite interesting to understand better how much of human thinking can be reasonably described as stochastic-parroting -- fairly clearly the amount isn't zero -- but I think it would be interesting as a finding in human psychology, not as a fact about LLMs.)
I am using stochastic parrot in the sense that since models are mainly trained on codebases of Latin characters, their coding strength reflects on the "goodness" of the codebases that were fed to them. More Latin codebases, better English coding performance. I am assuming from the paper, that when prompting in Chinese, the generated code is in Chinese for as much as possible (imported libs etc.).
Even if we didn't know french, it is straightforward to use deterministic tools (I.e. a dictionary) to parse the instruction, writing out the code in English, then refactor as many words as possible to french. The paper did not give any indication that this was happening (would be a novel result indeed) and so I would have to conclude that it is running like a stochastic parrot, more English codebase, better performance in English only.
I think this might still be in the realm of "maybe about as stochastic-parrot-y as human beings are"; I can easily imagine doing a worse job of remembering relevant stuff that happened to be in English if I had to do my work in some other language. Human memory is surprisingly context-dependent. But it would be interesting to see what happens if you ask an LLM to write code while talking to it in some language that has waaaaay less programming-related stuff on the internet. Swahili, perhaps. Is it much worse than when you prompt it in English, or a little worse, or what?
If the LLMs are very stochastic-parrot-y -- just piecing together bits of code associated with the words you wrote in English-or-Chinese-or-Swahili -- then I would expect them to get catastrophically worse when prompted in a language in which there's very little programming content on the internet. (For what it's worth, I think I also think this degree of stochastic-parrot-ness seems rather incompatible with what they are able to do. But others may disagree.) On the other hand, if they're more like humans -- somewhat better at remembering relevant things when they're in the same language as they're working in, etc., but operating at a conceptual level as well as pushing words around -- then I would expect the loss to be much more moderate.
(It seems somewhat relevant that the insides of a transformer network operate on embedding vectors rather than literal tokens; presumably those embedding vectors are much less language-specific.)
I think the demo font might not have Chinese characters at all. Most Latin fonts don't include CJK, and the system will fill in missing characters with available Chinese fonts. CJK fonts OTOH do include Latin characters.
As for tokenizers - I reckon that most of them are not optimized for CJK; they're good enough, and AI performance in CJK languages, so far, haven't been the top priority matter.
I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.
Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:
Language Relative tokens
--------------------------------------
English 1.00x
Portuguese 1.23x
Chinese (Simplified) 1.25x
German 1.31x
Spanish 1.32x
French 1.37x
Arabic 1.38x
Chinese (Traditional) 1.42x
Korean 1.47x
Swahili 1.49x
Hindi 1.57x
Japanese 1.66x
Burmese 3.16x
Amharic 5.78x
Santali 13.70x
Source: "Tokenizer Fairness in 2026", a reproduction/extension of
Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce
Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.
This made me realize that my usual practice of typing at 80% accuracy, full of mistakes, when sending input to an llm is probably increasing my input token count.
I'm always wondering if typos and grammar mistakes impact significantly the quality of the response. After all LLMs are next-token predictors, and I suspect that in the training set (internet), bad writing is correlated to low-quality content?
I was wondering about this one too.To make things worse I also use speech to text and that introduces its own inaccurate transcriptions and typos. Are there any reliable research around this ?
I think any of the current interfaces using thinking tokens and other contexts, make the actual input one types such a small part of the total input tokens that it probably doesn’t have much influence.
The kerning is absolutely awful in Safari. Looks fine in Chrome though...
This page reliably hangs Firefox 155 at 100% CPU for me.
I did most of my testing in Firefox 156 and it's fine for me
smooth as butter on 152 in windows 10, as long as we're all reporting
Yeah this site has a severe issue of some kind that I don't think I've ever seen before. NGL, I'm pretty impressed! It takes so much compute that scrolling lags, which feels very, very strange
Haha this did help me generate empathy for the assistant. Neat idea.
I wonder how this would look in Chinese Mandarin.
Completely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".
Perhaps relevant: Chinese language is not more efficient than English in vibe coding: A preliminary study on token cost and problem-solving rate https://arxiv.org/abs/2604.14210v1
This is interesting, thank you. It also reinforces the stocastic parrot theory of LLM. Maybe one mythical day when the majority of codebases around the world are written in CJK Han, vibe coding will become more efficient in Chinese
How does it say anything about the stochastic parrot theory of LLMs?
The only thing there that seems like it's relevant is the finding that the models they used had worse results when prompted in Chinese than when prompted in English. But ...
I am fairly sure I would do substantially worse work if I had to do it in French rather than in English, even if the work itself was all mathematics and programming and the like. Maybe that in some sense indicates that I am a stochastic parrot but it clearly doesn't indicate that I'm a stochastic parrot in some way worse than human beings are since I happen to be a human being myself.
... so what am I missing, that makes the models' worse performance in Chinese an indication that they're stochastic parrots in any interesting sense?
(I take it that "they're stochastic parrots just like we are" is not a very interesting sense. I mean, actually it would be quite interesting to understand better how much of human thinking can be reasonably described as stochastic-parroting -- fairly clearly the amount isn't zero -- but I think it would be interesting as a finding in human psychology, not as a fact about LLMs.)
I am using stochastic parrot in the sense that since models are mainly trained on codebases of Latin characters, their coding strength reflects on the "goodness" of the codebases that were fed to them. More Latin codebases, better English coding performance. I am assuming from the paper, that when prompting in Chinese, the generated code is in Chinese for as much as possible (imported libs etc.).
Even if we didn't know french, it is straightforward to use deterministic tools (I.e. a dictionary) to parse the instruction, writing out the code in English, then refactor as many words as possible to french. The paper did not give any indication that this was happening (would be a novel result indeed) and so I would have to conclude that it is running like a stochastic parrot, more English codebase, better performance in English only.
Aha, makes sense. Thanks.
I think this might still be in the realm of "maybe about as stochastic-parrot-y as human beings are"; I can easily imagine doing a worse job of remembering relevant stuff that happened to be in English if I had to do my work in some other language. Human memory is surprisingly context-dependent. But it would be interesting to see what happens if you ask an LLM to write code while talking to it in some language that has waaaaay less programming-related stuff on the internet. Swahili, perhaps. Is it much worse than when you prompt it in English, or a little worse, or what?
If the LLMs are very stochastic-parrot-y -- just piecing together bits of code associated with the words you wrote in English-or-Chinese-or-Swahili -- then I would expect them to get catastrophically worse when prompted in a language in which there's very little programming content on the internet. (For what it's worth, I think I also think this degree of stochastic-parrot-ness seems rather incompatible with what they are able to do. But others may disagree.) On the other hand, if they're more like humans -- somewhat better at remembering relevant things when they're in the same language as they're working in, etc., but operating at a conceptual level as well as pushing words around -- then I would expect the loss to be much more moderate.
(It seems somewhat relevant that the insides of a transformer network operate on embedding vectors rather than literal tokens; presumably those embedding vectors are much less language-specific.)
I think the demo font might not have Chinese characters at all. Most Latin fonts don't include CJK, and the system will fill in missing characters with available Chinese fonts. CJK fonts OTOH do include Latin characters.
As for tokenizers - I reckon that most of them are not optimized for CJK; they're good enough, and AI performance in CJK languages, so far, haven't been the top priority matter.
I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.
Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:
Source: "Tokenizer Fairness in 2026", a reproduction/extension of Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.
https://github.com/partyfly/tokenizer-fairness-2026
I typed the first stanza of 'Jabberwocky'.
Worked about as well as expected :D
This made me realize that my usual practice of typing at 80% accuracy, full of mistakes, when sending input to an llm is probably increasing my input token count.
Maybe spell checking for prompt text box would save a few bucks
I'm always wondering if typos and grammar mistakes impact significantly the quality of the response. After all LLMs are next-token predictors, and I suspect that in the training set (internet), bad writing is correlated to low-quality content?
I was wondering about this one too.To make things worse I also use speech to text and that introduces its own inaccurate transcriptions and typos. Are there any reliable research around this ?
I think any of the current interfaces using thinking tokens and other contexts, make the actual input one types such a small part of the total input tokens that it probably doesn’t have much influence.
Now I'm considering how many tokens I've wasted writing "tihs" instead of "this" - tens! Any how much extra work goes into understanding misspellings?