utiliD: A library with absolutely no dependencies for bare-metal programming and bootstrapping other D libraries

[Mike Franklin]

On Saturday, 11 May 2019 at 05:39:12 UTC, H. S. Teoh wrote:

> So potentially a D-based memcpy could have multiple concrete 
> implementations (copying strategies) that are statically chosen 
> based on the properties of T, like alignment and size.

Exactly.

> [...]
>> However, DMD won't do the right thing.
>
> Honestly, at this point I don't even care.

Personally I'd be fine with just killing of DMD's backend and 
just investing in LDC and GDC, but I don't think that's going to 
happen, and because of that, we have to care.  DMD is where 
policy and precedent is set for D.  To influence the direction of 
D, it must be done throught DMD.

> It could be possible to select multiple different memcpy 
> implementations by statically examining the properties of T.  I 
> think that might be one advantage D could have over just 
> calling libc's memcpy.  But you have to be very careful not to 
> outdo the compiler's optimizer so that it doesn't recognize it 
> as memcpy and fails to apply what would otherwise be a routine 
> optimization pass.

I understand.  That's why I'm calling it an "exploration" at this 
time.  I want to see what can and can't be done.

> At a certain point, this just begs the question "should I just 
> let the compiler's backend do its job by telling it plainly 
> that I mean memcpy, or should I engage in asm-hackery because 
> I'm confident I can outdo the compiler's codegen?".

I get that, but DMD is not the kind of backend that does that 
stuff.  If I could rely on DMD's, LDC's, and GDC's backend to 
just insert an optimized compiler intrinsic, without the C 
standard library, I would just leverage that. But that doesn't 
seem to be the world we're currently in.

> One thing that might be worth considering is for the *compiler* 
> to expose a memcpy intrinsic, and then let the compiler decide 
> how best to implement it (using its intimate knowledge of the 
> target machine arch), rather than trying to do it manually in 
> library code.

I would love for the backends to just know how to copy memory 
efficiently for all of their targets without me having to do 
anything, and without linking in the C standard library, but 
that's not what I'm seeing from the compilers right now.

> Based on what Andrei has voiced, the way to go would be to 
> merge Phobos and druntime into one, by making Phobos completely 
> opt-in so that you don't pay for what you don't use from the 
> heavier / higher-level parts of Phobos.  At a certain point it 
> becomes clear that the division between Phobos and druntime is 
> artificial, the result of historical accident, and not a 
> logical necessity that we have to keep. If Phobos is made 
> completely pay-as-you-go, the distinction becomes completely 
> irrelevant and the two might as well be merged into one.

Yes, but is making Phobos pay-as-you-go a real possibility?  I 
don't see it that way because all of Phobos has been developed 
under the assumption that all language features are implemented 
and available.  utiliD would be usable in an environment where 
only a subset of D's language features are available.

Also, Phobos has been developed under the assumption that any 
module in Phobos or druntime can be utilized as a dependency in 
any other module.  That has created a dependency mess in Phobos 
and I don't see how that can be disentangled without breaking 
everyone's code.  Furthermore, there is no clear hierarchy in 
Phobos where it is clear at the API level what language features 
are required for each module/function/whatever.  With utiliD, it 
is much clearer where the line is drawn in the hierarchy of 
language features.  Phobos will never be pay-as-you-go if you 
can't see what you're paying for as you go.

> See, this trouble is caused by the artificial boundary between 
> Phobos and druntime.  We should look into breaking down this 
> barrier, not enforcing it.

I agree.  We could actually merge druntime and Phobos into a 
single library today.  I also find the divide between Phobos and 
druntime artificial, but my goal with utiliD is different. I'm 
trying to create a library that does not require runtime language 
features.  I'm not proposing an artificial division that 
currently exists.  I'm trying to build something equivalent to a 
stack, where you start at a very low level (utilid) and add 
layers of increasing capability.  That's not what we have with 
Phobos and druntime today.

> I think the logical goal is to make Phobos completely 
> pay-as-you-go. IOW, an actual *library*, as opposed to a 
> tangled hairball of dependencies that always comes with strings 
> attached (can't import one small thing without pulling in the 
> rest of the hairball). A library is supposed to be a set of 
> resources which you can draw from as needed. Pulling out one 
> book (module) should not require pulling out half the library 
> along with it.

I agree, but that hairball is exactly what Phobos is right now. I 
don't see any way to start from that mess and achieve the 
pay-as-you-go opt-in continuum.  In a way, I'm starting over with 
utiliD, but I believe there is still value in druntime and Phobos 
that can be salvaged to start building an opt-in, pay-as-you-go 
stack of increasing features, sophistication, and capability in 
D, where you know, by what you're importing, what you're getting 
and what it costs.

Mike