Bartosz Milewski Missing post

[Andrei Alexandrescu]

Bartosz Milewski wrote:
> Andrei Alexandrescu Wrote:
> 
>> BCS wrote:
>>> Everything is indicating that shared memory multi-threading is
>>> where it's all going.
>> That is correct, just that it's 40 years late. Right now everything
>> is indicating that things are moving *away* from shared memory.
>> 
>> Andrei
> 
> I understand where you stand. You are looking at where the
> state-of-the-art hardware is going and that makes perfect sense.
> There is, however, a co-evolution of hardware and software that is
> not a simple software-follow-hardware (remember the era of RISC
> processors?).

I understand that. However, I don't understand how the comment applies 
to the situation at hand.

> I'm looking at programming languages and I don't see that
> away-from-shared-memory trend--neither in mainstream languages, nor
> in newer languages like Scala, nor in the operating systems.

Scala doesn't know what to do about threads. The trend I'm seeing is 
that functional languages are getting increasing attention, and that's 
exactly because they never share mutable memory. As far as I can see, 
languages that are based on heavy shared mutation are pretty much dead 
in the water. We have the chance to not be so.

> There
> are many interesting high-level paradigms like message passing,
> futures, actors, etc.; and I'm sure there will be more in the future.
> D has a choice of betting the store on one of these paradigms (like
> ML did on message passing or Erlang on actors) or to try, build solid
> foundations for a multi-paradigm language or do nothing. I am trying
> to build the foundations.

Building foundations is great. What I'm seeing, however, is one very 
heavy strong pillar put in a place that might become the doghouse. I'm 
not at all sure the focus must be put on high-level race avoidance, 
particularly given that the cost in perceived complexity is this high.

> The examples that I'm exploring in my posts are data structures that
> support higher level concurrency: channels, message queues, lock-free
> objects, etc. I want to build a system where high-level concurrency
> paradigms are reasonably easy to implement.
> 
> Let's look at the alternatives.
> 
> Nobody thinks seriously of making message passing a language feature
> in D. The Erlangization of D wouldn't work because D does not provide
> guarantees of address space separation (and providing such guarantees
> would cripple the language beyond repair).

Why wouldn't we think of making message passing a language feature in D? 
Why does message passing need erlangization to support message passing?

> Another option we discussed was to provide specialized, well tested
> message queues in the library and count on programmers' discipline
> not to stray away from the message passing paradigm.

That's not what I discussed. I think there is is an interesting point 
that you've been missing, so please allow me to restate it.

What I discussed was a holistic approach in which language + standard 
library provides a trusted computing base. Consider Java's new (for 
arrays) and C's malloc. The new function cannot be defined in Java 
because it would require unsafe manipulation underneath, so it is 
defined by its runtime support library. That runtime support is 
implemented in the likes of C. However, due to the fact that the runtime 
support is part of Java, in fact Java does have dynamic memory 
allocation - and nobody blinks an eye.

C has famously had a mantra of self-sufficiency: its own support 
libraries have been written in C, which is pretty remarkable - and 
almost unprecedented at the time C was defined. For example, C's malloc 
is written in C, but (and here's an important detail) at some point it 
becomes _nonportable_ C. So even C has to cross a barrier of some sorts 
at some point.

How is this related to the discussion at hand? You want to put all 
concurrency support in the language. That is, you want to put enough 
power into the language to be able to typecheck a variety of concurrent 
programming primitives and patterns. This approach is blindsided to the 
opportunity of defining some of these primitives in the standard 
library, in unsafe/unportable D, yet offering safe primitives to the 
user. In the process, the user is not hurt because she still has access 
to the primitives. What she can't do is define their own primitives in 
safe D. But I think that's as useless a pursuit as adding keywords to C 
to allow one to implement malloc() in safe C.

> Without
> type-system support, though, such queues would have to be either
> unsafe (no guarantee that the client doesn't have unprotected aliases
> to messages), or restrict messages to very simple data structures
> (pass-by-value and maybe some very clever unique pointer/array
> template). The latter approach introduces huge complexity into the
> library, essentially making user-defined extensions impossible
> (unless the user's name is Andrei ;-) ).

Complexity will be somewhere. The problem is, you want to put much of it 
in the language, and that will hit the language user too. Semantic 
checking will always be harder on everyone than a human being who sits 
down and implements a provably safe library in ways that the compiler 
can't prove.

> Let's not forget that right now D is _designed_ to support
> shared-memory programming. Every D object has a lock, it supports
> synchronized methods, the keyword "shared" is being introduced, etc.
> It doesn't look like D is moving away from shared memory. It looks
> more like it's adding some window dressing to the pre-existing mess
> and bidding its time. I haven't seen a comprehensive plan for D to
> tackle concurrency and I'm afraid that if D doesn't take a solid
> stance right now, it will miss the train.

I think we can safely ditch this argument. Walter had no idea what to do 
about threads when he defined D, so he put whatever he saw and 
understood in Java. He'd be the first to say that - actually, he 
wouldn't tell me so, he _told_ me so.

To me, adding concurrency capabilities to D is nothing like adding 
window dressing on top of whatever crap is there. Java and C++ are in 
trouble, and doing what they do doesn't strike me as a good bet. You're 
right about missing the train, but I think you and I are talking about 
different trains. I don't want to embark on the steam-powered train.


Andrei