Proposed Phobos equivalent of wcswidth()

[Dmitry Olshansky]

On Tuesday, 16 January 2018 at 23:01:19 UTC, H. S. Teoh wrote:
> On Tue, Jan 16, 2018 at 05:49:11PM +0000, Dmitry Olshansky via 
> Digitalmars-d wrote:
>> On Monday, 15 January 2018 at 19:52:07 UTC, H. S. Teoh wrote:
> [...]
>> > One thing I'm seeking help with, and this is mainly directed 
>> > at Dmitry Olshansky but can be anyone here who knows the 
>> > internal workings of std.uni well enough, is how to 
>> > transform the Trie generated by the static ctor into 
>> > compile-time TrieNode declarations.  This is one blocker for 
>> > my turning this code into a Phobos PR, because I don't want 
>> > to incur the cost of initializing this trie at runtime.
>> 
>> 
>> Checkout my horribly named repo gsoc-uni-benchmark:
>> 
>> https://github.com/DmitryOlshansky/gsoc-bench-2012/blob/master/gen_uni.d
>> 
>> This is what generates unicode tables.
>> Need to revise it, as folks were delicate enough to hand-patch
>> auto-generated code in Phobos.
>> 
>> Maybe make some of that user-acessible.
> [...]
>
> Whoa. There's some pretty cool stuff in there!  Thanks, I've 
> started experimenting with pre-generating the width table.  
> Pretty neat. There's a lot of hidden gems in std.uni that I 
> never knew existed, hidden away under `private`. :-D

The intent is to open that up somehow, to allow folks to make 
their own extended versions of std.uni. Unicode is all about 
“tailoring” - adjusting algorithm to your specific regional 
preferences hy messing with tables.

I think there is at least 1 bug in Bugzilla on this.
>
> One thing, though: I think it would benefit us all if we could 
> import at least gen_uni into Phobos, so that in the future when 
> we need to update std.uni to a new version of Unicode, it can 
> be (mostly) automated.  It's better to have the tools to 
> generate the tables in Phobos itself, than to be dependent on 
> an external repo that may go out-of-sync eventually.

Yes but it’s non-trivial at the moment, if you take a look at 
script to generate stuff it takes both 32-bit and 64-bit 
executables to populate tables.

I think having it in tools repo should be fine though. Last time 
I tried to update to Unicode 10, I found one table in Phobos that 
is missing from generator (ooops!).

>
> When I get around to making a PR for strwidth AKA displayWidth, 
> the plan is to check-in compileWidth.d in some form into Phobos 
> somewhere, so that somebody else can pick it up and improve the 
> implementation in the future if I'm not around / unavailable.
>
> If we can get gen_uni into Phobos, perhaps we can even include 
> the displayWidth table generation in gen_uni too, so that all 
> the table generation code is in one place.

Right. A good step would be to move it to tools, then add your 
code.


>
>
> T