Very limited shared promotion

[Ola Fosheim Grøstad]

On Wednesday, 19 June 2019 at 10:50:45 UTC, Jonathan M Davis 
wrote:
> I don't follow. What do you mean by obtain write access? If 
> you're talking about shared, scope doesn't care.

Yes, my formulation wasn't clear.  What I meant was "what would 
be the best way to write an OO-database interface where you can 
request read/write access to an object or a set of objects?".

 From what has been said in this thread I imagine that the 
database would hand out a "scope"-qualified reference of some 
sort that prevents the lock from lingering after the scope has 
been exited.

Is "shared" the basic mechanism that one ought to use to write 
robust thread safe APIs in D?

So, maybe you would get something like this when requesting 
objects from an in-memory database:

- Obtaining a write-locked-object: returned as a "scoped 
unshared" reference-type.

- Obtaining a read-locked-object: returned as a "scoped 
immutable" reference-type.


I am just trying to figure out what the provided mechanisms could 
be useful for, and what the missing bits might be.  What does it 
enable?  What does it not enable?  In terms of API design, static 
type checking and "dynamic reference counting of locks".

How does it affect the ability to call into libraries that was 
never written with multi-threading in mind?  What can go wrong if 
you do so? Etc.


> Actually, using scope when you don't need to would be a serious 
> problem. For instance, scope does _not_ interact well with 
> range-based code. The typical thing for a lot of range-based 
> functions to do is to take the argument, wrap it in a struct 
> that's a new range, and return it, whereas if the parameter 
> were scope, then you couldn't put the argument or any reference 
> to it in the struct.

Ah. But this could be resolved if "scope" kept count of the block 
level you are at?

You would just need an implicit scope-level count. So the level 
where a reference were turned into "scoped" would be level 0, 
whenever it is transferred to a new scope it would have +1 added 
to its level.


> Walter is against data flow analysis in most cases largely 
> because when language semantics depend on it, you have to 
> specify exactly how the data flow analysis works, and it's 
> never clear exactly how far you should go with it. He routinely 
> shoots down features that would require data flow analysis.

I think it might be possible to reuse existing code though, if 
you just generate a suitable graf from the internal 
representation in the compiler.

There is a generic theoretical conception for data flow analysis 
called Monotone Frameworks.

https://www.cs.cmu.edu/~aplatzer/course/Compilers/27-monframework.pdf


> I have no clue how feasible that would be. I'm also not sure 
> what benefit there would be in telling the compiler that a 
> piece of code guarantees something that the compiler doesn't 
> already check for.

The code gen has to know where the sequencing points are, but 
maybe it could do better with more information. I guess 
sequencing can be fixed by casting, but there might be other 
things you want to verify. E.g.: does references escape, are 
functions idempotent…


I believe the generic idea is the right one for a language like D:

There are many concurrency paradigms, it will be difficult to 
pick one over another since hardware keeps changing and use 
contexts are different.

So if you can move concurrency to libraries, and also provide 
crude verification of correctness (or at least reduce the number 
of common mistakes), then people can write different concurrency 
frameworks and experiment.

One problem with concurrent languages like Pony and Chapel, that 
assume a particular paradigm, is that they make assumptions about 
the hardware architecture.

It is not at all obvious that the 
unified-memory-with-manycores-architecture will win out over the 
next decade.  Event today that seems to be primarily a server 
architecture.

The current client/embedded architecture appears to be a small 
handfull of cores with many specialised hardware units with 
library support.

So I spot a diverging trend there. But it is hard to predict 
where the market will head in the next decade. Will the 
architecture be unified again by CPUs absorbing what 
co-processors do, or will it move more towards a heterogeneous 
distributed architecture (many independent units with local 
memory)?

D's focus on single threaded programming is fairly reasonable in 
terms of being able to adapt to many types of hardware in a way 
that fits how most programmers approach programming today.

So, it might be reasonable to figure out how to enable 
concurrency libraries that provide verification mechanisms (like 
escape analysis/guarantees) rather than special casing language 
features.


> It has been proposed in the past that folks wanted to use UDAs 
> to indicate stuff that was somehow then checked by the 
> compiler, but I don't know exactly what the expectation there 
> was. Certainly, there are bound to be things that we could do 
> along those lines, but a concrete proposal would be necessary.

Yes. So one should think about what kind of basic mechanisms one 
would want to be able to implement as std-library features. And 
what prevents that from being possible and effective/robust.


> Honestly, given how D's type system works, I don't think that 
> it's really possible to do anything substantially different 
> from C++ as far the built-in concurrency stuff goes.

Doing better than C++ is important though.  C++ was conceived 
before concurrency became pressing (C++ was designed at a time 
when multiple cores was prohibitively expensive), so concurrency 
is C++'s big weakness.  The concurrency shortcomings of C++ is 
also why people create languages like Chapel.  So, I'd have to 
agree with Manu that it is important to get this right, but it is 
also difficult to get right, so one should not rush it.


> considerable improvement over C++. Regardless, before we can be 
> sure of nicer features that we might be able to build on top of 
> shared, we first need to get the low level aspects sorted out.

One has to do both.  It is absolutely necessary to specify all 
the use cases that should be present in a solid concurrency 
standard library before deciding on the shared semantics.

D has to break out of the evolutionary type system design cycle 
so that the type system design does not get amended with special 
casing and counter-intuitive exceptions after release.


Ola.