Reimplementing the bulk of std.meta iteratively

[H. S. Teoh]

On Tue, Sep 29, 2020 at 06:43:23PM +0000, bachmeier via Digitalmars-d wrote:
> On Tuesday, 29 September 2020 at 14:07:14 UTC, Andrei Alexandrescu wrote:
[...]
> > There is no doubt to Walter or myself that the way built-in AAs were
> > done in D is a resounding failure.
> 
> If so, it's a pretty good resounding failure, because I like them and
> they would most likely not be available to me now if they hadn't been
> built into the language. [...]

Same here.  Built-in AA's were one of the big reasons that drew me to D.
Now granted, the way they are implemented perhaps leaves a lot to be
desired, but having built-in support for AA's IMO is a fundamental
language feature.  Whether it's implemented in the compiler or the
standard library is secondary, but it's not something that should be
left to an external 3rd party library.

It would definitely be a detriment if D had followed the same path as
C++ in having hash tables in the standard library (extremely late in the
history of C++, I might add -- something I will probably hold forever
against C++ :-P), but requiring a lot of manual effort to use: no
default hash function for built-in types like structs or ints, need to
manually declare all sorts of things just to instantiate a lousy
hashtable, having inordinately long derived type names just to name
something that in D is as simple as `MyStruct[string]`, and just general
needless gratuitous barriers to usage.

Some level of language support is necessary to grease all these
usability gears: at the very least the language must support nice, easy
syntax to declare AA's -- even if the actual implementation is delegated
to a library type in the standard library.


[...]
> > The way they should have been done was a templated library type
> > AssociativeArray!(K, V) with the V[K] type syntax sugar on top. That
> > would have allowed the implementation to improve along with the rest
> > of the language and its library.
> 
> Does anything prevent the introduction of a new library solution? If
> it's good enough, it can eventually be added to the language.
[...]

A lot of effort has been put in to move the AA implementation into
object.d, and the current implementation has also undergone several
rounds of improvement, code quality wise and performance wise.  But
there remain some magic bits that currently cannot be duplicated in
"user space", so to speak, (as opposed to compiler space).

One is the handling of const/immutable on keys, which, in spite of being
magically handled by the compiler, nevertheless suffers from flaws: to
prevent key aliasing problems that cause inconsistencies in the AA, as
of several releases ago AA keys are now implicitly `const`. However,
this does not actually solve the problem -- because mutable can
implicitly convert to const, so the aliasing problem still exists. It
would be nice to make it immutable, but that breaks existing code in
some places, and in any case switching from const to immutable isn't
something that can be done as a gradual deprecation -- it's all or
nothing.  But regardless of the correctness issue, the handling of
const/immutable qualifiers on AA keys depends on compiler magic, and
currently either isn't expressible in library code, or else is so
cumbersome it's simply impractical.

Another is the magic handling of:

	Data[string][string] aa;
	aa["abc"]["def"] = Data(...);

For this, we need opIndexCreate, which we currently don't have:

	https://issues.dlang.org/show_bug.cgi?id=7753

There may be one or two other issues that I can't recall off the top of
my head, but basically, we've come a long way since the original AA
implementation, and now only need to remove a few more magical barriers
before a fully-library AA solution is possible in druntime.

//

Of course, all of this confirms what Andrei is saying about too much
magic being boiled into the compiler: it can do stuff library code
cannot, and because of that, fixing problems in AAs requires digging
deep into compiler innards.  Worse yet, some of the rules governing AA
magic in the compiler isn't fully compatible with the rules library code
is subjected to; and trying to replicate this behaviour in library code
is very painful, or outright impossible.


T

-- 
Doubt is a self-fulfilling prophecy.