I'm Chinese American, and I don't agree with your statement that string libraries are Anglophile. How would you encapsulate the 10,000 commonly used Chinese characters? It's just the reality of having a lot of characters in a language. Not much else you can do to speed up processing. How would you design string storage to be faster for a language like Chinese?
English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience.
Let me ask you, with 10k commonly used characters doesn't that lead to shorter texts? Kind of like how higher base numbers can encode larger numbers with fewer digits, in that case the longer encoding of UTF-8 could be made up for by using fewer characters. Or am I wrong about this assumption?
As an example, suppose that there are one character that denotes the word 'house', if that single character is encoded using five bytes it takes the same amount of space as the english encoding.
I won't pretend to have a solution. I guess you would like to have some compression scheme, since I'm guessing it would save space over having 2-4 bytes (however many there are in the Chinese language) for every character. You won't gain much compared to having an English-centric encoding scheme, I guess, in the case of a language with a large amount of characters.
But it's still funny to me how even the computer who speaks in 1's and 0's favours English-centered notation.
> English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience.
Pragmatism ũber alles, chants the American. I guess I'm not impressed by the support of non-English languages in IT.
I have for that matter met engineering students who don't seem to speak a lick of English, maybe even people studying CS/CE.
See, here is the thing. At the end of the day, hypotheticals are worthless, concrete solutions are all that matters. It isn't anglocentricism that we picked the solution that is actually fleshed out and works over the vague hypothetical solution. It's "get-shit-done"-ism
> I guess you would like to have some compression scheme, since I'm guessing it would save space over having 2-4 bytes (however many there are in the Chinese language) for every character.
You can of course compress a text[1] after encoding it, but that really is an unrelated topic. You can't get 10k possible characters into 8 bits, you need to go multi-byte.
> Pragmatism ũber alles, chants the American.
"Un bon mot ne prouve rien." -Voltaire
[1] _most_ text that you will see in practice. No lossless compression algorithm can compress _any_ possible text.
> See, here is the thing. At the end of the day, hypotheticals are worthless, concrete solutions are all that matters. It isn't anglocentricism that we picked the solution that is actually fleshed out and works over the vague hypothetical solution. It's "get-shit-done"-ism
"If you don't know of a solution yourself, shut up." Similar to "if you can't play guitar as well as <a player>, you don't get to have an opinion".
Admittedly in this context I might as well have thought I had something better to offer, given my original post. But as I've said, I don't. It was more of a historical note. I don't see how, given an alternative history, computers wouldn't favour for example the Russian alphabet.
And while we're at it, you might lecture me on how text/ASCII-centered protocols are superior to a binary format. Because I honesetly don't know.
And the fact that IT is Anglo centric goes way beyond Shannon entropy.
"If you don't know of a solution yourself, shut up." Similar to "if you can't play guitar as well as <a player>, you don't get to have an opinion".
I'm not telling you to shut up. I am telling you to not act offended that a tangible working solution was chosen over a hypothetical solution. In other words, don't act like the universe is unfair because Paul McCartney is famous for songwriting while you are not, even though you totally could have hypothetically written better songs.
> "I don't see how, given an alternative history, computers wouldn't favour for example the Russian alphabet."
In an alternative universe where CP1251 was picked as the basis of the first block in Unicode instead of ASCII, it would have been for the same reasons that ASCII was picked in this universe.
In that universe, you'd just be complaining that Unicode was Russo-centric.
What reason, in this universe, would there have been to go that route?
> Compressing as in something like Huffman encoding. Maybe I was misusing the names.
Huffman encoding is a method used for lossless compression of particular texts. It does not let you put more than 256 characters into a single byte in a character encoding.
The guys that made JIS X 0212 were not missing something when they made JIS X 0208, a two byte encoding, prior to Unicode.
> And the fact that IT is Anglo centric goes way beyond Shannon entropy.
Okay. Complain about instances where it actually exists, and in discussions where it is actually relevant.
> In that universe, you'd just be complaining that Unicode was Russo-centric.
Yes, obviously.
> It does not let you put more than 256 characters into a single byte in a character encoding.
Which I have never claimed. (EDIT: I think we're talking past each other: my point was that things like Huffman encoding encodes the most frequent data with the lowest amount of bits. I don't know how UTF-8 is implemented, but it seems conceptually similar. There is a reason that I didn't want to get anywhere near the nitty-gritty of this.)
> Okay. Complain about instances where it actually exists, and in discussions where it is actually relevant.
A character coding has an equal distribution of each code point. Each code point is represented once.
"For a set of symbols with a uniform probability distribution and a number of members which is a power of two, Huffman coding is equivalent to simple binary block encoding, e.g., ASCII coding."
Huffman encoding something written in Japanese is useful. It is not useful for creating a Japanese character set.
Get it?
If you don't buy it, then try it on pen and paper. Imagine a hypothetical 10-character alphabet, and try to devise an encoding that will let you fit it into a two-bit word, without going multi-word. Use prefix codes or whatever you want.
It's not going to happen. You also aren't going to get 10k characters into an 8-bit/word single-word character set.
English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience.