[OT] Effect of UTF-8 on 2G connections

[Joakim via Digitalmars-d]

On Wednesday, 1 June 2016 at 14:58:47 UTC, Marco Leise wrote:
> Am Wed, 01 Jun 2016 13:57:27 +0000
> schrieb Joakim <dlang at joakim.fea.st>:
>
>> No, I explicitly said not the web in a subsequent post.  The 
>> ignorance here of what 2G speeds are like is mind-boggling.
>
> I've used 56k and had a phone conversation with my sister while 
> she was downloading a 800 MiB file over 2G. You just learn to 
> be patient (or you already are when the next major city is 
> hundreds of kilometers away) and load only what you need. Your 
> point about the costs convinced me more.

I see that max 2G speeds are 100-200 kbits/s.  At that rate, it 
would have taken her more than 10 hours to download such a large 
file, that's nuts.  The worst part is when the download gets 
interrupted and you have to start over again because most 
download managers don't know how to resume, including the stock 
one on Android.

Also, people in these countries buy packs of around 100-200 MB 
for 30-60 US cents, so they would never download such a large 
file.  They use messaging apps like Whatsapp or WeChat, which 
nobody in the US uses, to avoid onerous SMS charges.

> Here is one article spiced up with numbers and figures: 
> http://www.thequint.com/technology/2016/05/30/almost-every-indian-may-be-online-if-data-cost-cut-to-one-third

Yes, only the middle class, which are at most 10-30% of the 
population in these developing countries, can even afford 2G.  
The way to get costs down even further is to make the tech as 
efficient as possible.  Of course, much of the rest of the 
population are illiterate, so there are bigger problems there.

> But even if you could prove with a study that UTF-8 caused a
> notable bandwith cost in real life, it would - I think - be a
> matter of regional ISPs to provide special servers and apps
> that reduce data volume.

Yes, by ditching UTF-8.

> There is also the overhead of
> key exchange when establishing a secure connection:
> http://stackoverflow.com/a/20306907/4038614
> Something every app should do, but will increase bandwidth use.

That's not going to happen, even HTTP/2 ditched that requirement. 
  Also, many of those countries' govts will not allow it: google 
how Blackberry had to give up their keys for "secure" BBM in many 
countries.  It's not just Canada and the US spying on their 
citizens.

> Then there is the overhead of using XML in applications
> like WhatsApp, which I presume is quite popular around the
> world. I'm just trying to broaden the view a bit here.

I didn't know they used XML.  Googling it now, I see mention that 
they switched to an "internally developed protocol" at some 
point, so I doubt they're using XML now.

> This note from the XMPP that WhatsApp and Jabber use will make
> you cringe: https://tools.ietf.org/html/rfc6120#section-11.6

Haha, no wonder Jabber is dead. :) I jumped on Jabber for my own 
messages a decade ago, as it seemed like an open way out of that 
proprietary messaging mess, then I read that they're using XML 
and gave up on it.

On Wednesday, 1 June 2016 at 15:02:33 UTC, Wyatt wrote:
> On Wednesday, 1 June 2016 at 13:57:27 UTC, Joakim wrote:
>>
>> No, I explicitly said not the web in a subsequent post.  The 
>> ignorance here of what 2G speeds are like is mind-boggling.
>>
> It's not hard.  I think a lot of us remember when a 14.4 modem 
> was cutting-edge.

Well, then apparently you're unaware of how bloated web pages are 
nowadays.  It used to take me minutes to download popular web 
pages _back then_ at _top speed_, and those pages were a _lot_ 
smaller.

> Codepages and incompatible encodings were terrible then, too.
>
> Never again.

This only shows you probably don't know the difference between an 
encoding and a code page, which are orthogonal concepts in 
Unicode.  It's not surprising, as Walter and many others 
responding show the same ignorance.  I explained this repeatedly 
in the previous thread, but it depends on understanding the tech, 
and I can't spoon-feed that to everyone.

>> Well, when you _like_ a ludicrous encoding like UTF-8, not 
>> sure your opinion matters.
>
> It _is_ kind of ludicrous, isn't it?  But it really is the 
> least-bad option for the most text.  Sorry, bub.

I think we can do a lot better.

>>> No. The common string-handling use case is code that is 
>>> unaware which script (not language, btw) your text is in.
>>
>> Lol, this may be the dumbest argument put forth yet.
>
> This just makes it feel like you're trolling.  You're not just 
> trolling, right?

Are you trolling?  Because I was just calling it like it is.

The vast majority of software is written for _one_ language, the 
local one.  You may think otherwise because the software that 
sells the most and makes the most money is internationalized 
software like Windows or iOS, because it can be resold into many 
markets.  But as a percentage of lines of code written, such 
international code is almost nothing.

>> I don't think anyone here even understands what a good 
>> encoding is and what it's for, which is why there's no point 
>> in debating this.
>
> And I don't think you realise how backwards you sound to people 
> who had to live through the character encoding hell of the 
> past.  This has been an ongoing headache for the better part of 
> a century (it still comes up in old files, sites, and systems) 
> and you're literally the only person I've ever seen seriously 
> suggest we turn back now that the madness has been somewhat 
> tamed.

No, I have never once suggested "turning back."  I have suggested 
a new scheme that retains one technical aspect of the prior 
schemes, ie constant-width encoding for each language, with a 
single byte sufficing for most.  _You and several others_, 
including Walter, see that and automatically translate that to, 
"He wants EBCDIC to come back!," as though that were the only 
possible single-byte encoding and largely ignoring the 
possibilities of the header scheme I suggested.

I could call that "trolling" by all of you, :) but I'll instead 
call it what it likely is, reactionary thinking, and move on.

> If you have to deal with delivering the fastest possible i18n 
> at GSM data rates, well, that's a tough problem and it sounds 
> like you might need to do something pretty special. Turning the 
> entire ecosystem into your special case is not the answer.

I don't think you understand: _you_ are the special case.  The 5 
billion people outside the US and EU are _not the special case_.  
Yes, they have not mattered so far, because they were too poor to 
buy computers.  But the "computers" with the most sales these 
days are smartphones, and Motorola just launched their new Moto 
G4 in India and Samsung their new C5 and C7 in China.  They 
didn't bother announcing release dates for these mid-range 
phones- well, they're high-end in those countries- in the US.  
That's because "computer" sales in all these non-ASCII countries 
now greatly outweighs the US.

Now, a large majority of people in those countries don't have 
smartphones or text each other, so a significant chunk of the 
minority who do buy mostly ~$100 smartphones over there can 
likely afford a fatter text encoding and I don't know what 
encodings these developing markets are commonly using now.  The 
problem is all the rest, and those just below who cannot afford 
it at all, in part because the tech is not as efficient as it 
could be yet.  Ditching UTF-8 will be one way to make it more 
efficient.

On Wednesday, 1 June 2016 at 16:15:15 UTC, Patrick Schluter wrote:
> Indeed, Joakim's proposal is so insane it beggars belief (why 
> not go back to baudot encoding, it's only 5 bit, hurray, it's 
> so much faster when used with flag semaphores).

I suspect you don't understand my proposal.

> As a programmer in the European Commission translation unit, 
> working on the probably biggest translation memory in the world 
> for 14 years, I can attest that Unicode is a blessing. When I 
> remember the shit we had in our documents because of the code 
> pages before most programs could handle utf-8 or utf-16 (and 
> before 2004 we only had 2 alphabets to take care of, Western 
> and Greek). What Joakim does not understand, is that there are 
> huge, huge quantities of documents that are multi-lingual.

Oh, I'm well aware of this.  I just think a variable-length 
encoding like UTF-8 or UTF-16 is a bad design.  And what you have 
to realize is that most strings in most software will only have 
one language.  Anyway, the scheme I sketched out handles multiple 
languages: it just doesn't optimize for completely random jumbles 
of characters from every possible language, which is what UTF-8 
is optimized for and is a ridiculous decision.

> Translators of course handle nearly exclusively with at least 
> bi-lingual documents. Any document encountered by a translator 
> must at least be able to present the source and the target 
> language. But even outside of that specific population, 
> multilingual documents are very, very common.

You are likely biased by the fact that all your documents are 
bilingual: they're _not_ common for the vast majority of users.  
Even if they were, UTF-8 is as suboptimal, compared to the 
constant-width encoding scheme I've sketched, for bilingual or 
even trilingual documents as it is for a single language, so even 
if I were wrong about their frequency, it wouldn't matter.