Very limited shared promotion

[Ola Fosheim Grøstad]

On Wednesday, 19 June 2019 at 08:45:10 UTC, Jonathan M Davis 
wrote:
> scope doesn't care where the object is stored. If something is 
> scope, then you can't take any references to it in @safe code. 
> The whole point is to ensure that no references escape. Whether 
> the objects are GC-allocated are not is irrelevant.

So, the same as this:
https://dlang.org/spec/function.html#return-scope-parameters

Does this mean then that when you obtain write access to an 
object that "write access reference" will be scope qualified?

And then you cannot use that reference with any function that 
does not have scope qualified parameters?

So basically all library code has to add "scope" to all its 
parameters whether it is written with shared in mind or not.

So, "scope" becomes like "pure". Something that ought to be the 
default, but has to be added manually to all function prototypes 
in order to make safe multi-threaded programming less annoying.

Or will the compiler automatically deduce that a function 
parameter is fulfilling the "scope" requirements even when it has 
not been specified?

I guess it should, otherwise you'll end up with the "const" 
transition in C++, where you had to do ugly cast-hacks when 
calling functions that did treat parameters as const but the 
function signature had not specified it.

> implicitly cast to or from shared. As it is, in general, D 
> can't even know if there are multiple references to the same 
> object. Without that kind of information, it's not even 
> possible to know whether it's safe to pass an object from one 
> thread to another.

You can do this with dataflow in many cases, maybe in most useful 
cases. Although something like unique_ptr in C++ helps (by 
convention).

> And without some understanding of which concurrency primitives 
> are being used to protect an object or group of objects, it 
> can't do something as simple as know that locking a mutex has 
> actually protected those objects such that they can safely be 
> manipulated from the current thread.

Right. But you could do something dynamic with reference counting 
and "framework conventions".

e.g..:

auto dblock = database.allocate_lock();
auto obj_ref_lockcounting = dblock.require_write_access(obj_id);

Then maybe you just need some metaprogramming mechanisms to 
inform the type system what the framework 
"reference_counting_lock_object" does, so that the type system 
knows what objects have been locked and that when dblock goes out 
of scope then all locks have been released.

Seems plausible, but I haven't given it much thought.

> be necessary to make that happen, because it would require 
> significant changes to how D works.

Maybe, although I wonder how much you could do as library code. 
So I don't know.

If metaprogramming is going to be the main focus of D then it 
makes sense to focus on allowing library authors to inform the 
type system of what their library types are capable of. Basically 
the opposite of a traditional type system where the typesystem 
checks constraints.

So, the basic idea would be that the library tells the type 
system "my code provide these guarantees", rather than the the 
library asking the type system to verify that a set of guarantees 
hold.

> casting. _Maybe_, there will be a few, small places like with 
> what Manu is suggesting here where we will be able to leverage 
> what the compiler knows to implicitly convert something, but in 
> general, that's really not going to work.

I think that make such special cases is a bit dangerous. It is 
really going down the path of C++ that leads to a situation where 
you only appeal to the same audience as C++.

It also leaves the field open to Rust. I agree that borrowing 
might be tedious for memory safety, but might be generally 
suitable for making safer multi-threading available to more 
programmers. At least on the surface level, but I haven't spent a 
lot of time thinking about the limitations of borrow-checking in 
relation to multi-threading. So, it is just a hunch.

I think it would be a mistake to outright dismiss tracking 
references.  There might be libraries available that can do it 
for you even, e.g. provide a compiler switch that enables the 
compiler to generate a graph that is passed onto an external 
library that does the checking.

> I'd be very surprised if anything like that were considered 
> acceptable for D. When stuff like Rust's borrowing has been 
> brought up before, Walter and Andrei have made it pretty clear 
> that we're not going to do something like that with D's type 
> system.

Ok, but there are two issues here:

1. What you can query the type system for.

This has to be available at all times and has to be tightly 
integrated into the type system. There is a high cost to adding 
such features.

2. Type annotations that enable verification.

This does not have to be tightly integrated into the type system. 
You can provide it as a separate module with clear boundaries. 
And you might even be able to turn it off for faster 
recompilation.
There is a low cost to adding such features.


> Even getting something like DIP 1000 has been a major ordeal, 
> and I don't think that it would have ever happened unless 
> Walter had been convinced that it was absolutely needed for 
> @safe code to be able to do stuff like reference counting.

DIP1000 looks too limiting, but I don't know what it has enabled 
and where it falls short in practice.

I guess strength and weaknesses will show up as people build new 
frameworks around it.

> shared exists so that almost everything in D can be treated as 
> thread-local, and the type system prevents you from converting 
> between thread-local and shared without casting. Even immutable 
> ends up as part of it, because it's implicitly shared. The 
> compiler knows which objects are thread-local and which are 
> shared and can use that information in code generation.

Ok, so the basic idea is to allow D to shuffle instructions/code 
more around because the compiler can assume that nothing held in 
registers is ever stale.

And "shared" was just added to push "everything multi-threaded" 
into the future. So, now you have to figure out what that means 
in order to make it more useful to people who do more advanced 
programming in D (like Manu).

> force the matter. @system/@trusted then gets involved in the 
> sense that the cast is @system, but beyond that, @safety 
> doesn't have anything to do with it. I don't see how any of 
> that could be done via a library rather than the type system.

Well, it is currently just a question. And in my view the answer 
is not known.

I think there might be options to do it if we change your view on 
what a type system is from:

1. The usage of the type system specifies what the compiler 
should verify.

to

2. Vetted libraries can inform the type system of what it is 
capable of.


Ola.