Bartosz Milewski Missing post

[Andrei Alexandrescu]

Bartosz Milewski wrote:
> Andrei Alexandrescu Wrote:
>> Scala doesn't know what to do about threads.
> 
> That's my impression too, although Scala's support for actors leaves
> D in the dust.

Scala actors are a library.

>> The trend I'm seeing is that functional languages are getting
>> increasing attention, and that's exactly because they never share
>> mutable memory. As far as I can see, languages that are based on
>> heavy shared mutation are pretty much dead in the water. We have
>> the chance to not be so.
>> 
> 
> It's a very sweeping statement. I just looked at the TIOBE index and
> couldn't find _any_ functional languages in the top 20.

We can safely ditch Tiobe, but I agree that functional languages aren't 
mainstream. There are two trends though. One is that for most of its 
existence Haskell has had about 12 users. It has definitely turned an 
exponential elbow during the recent years. Similar trends are to be seen 
for ML, Ocaml, and friends.

The other trend is that all of today's languages are scrambling to add 
support for pure functional programming.

>> I'm not at all sure the focus must be put on high-level race
>> avoidance, particularly given that the cost in perceived complexity
>> is this high.
> 
> The complexity argument is tenuous. It might look like my proposal is
> complex because I'm dropping the whole system in at once. But it's a
> solid, well thought-out system. What I see happening in D is the
> creeping complexity resulting from sloppy design. We've been talking
> for years about closing some gaping holes in the design of arrays,
> slices, immutable, qualifier polymorphism--the list goes on--and
> there's little progress. There is no solid semantics for scope and
> shared. A solid solution to those issues will look complex too.

What do those holes have to do with the problem at hand? I'm seeing 
implementation bugs, not design holes. I'd love them to be fixed as much 
as the next guy, but I don't think we're looking at issues that would be 
complex (except for scope which sucks; I never claimed there was a 
solution to that). All of other features you mention have no holes I 
know of in their design.

>> Why wouldn't we think of making message passing a language feature
>> in D?
> 
> Because we don't have even a tiniest proposal describing it, not to
> mention a design.

That doesn't mean we shouldn't think of it.

>> Why does message passing need erlangization to support message
>> passing?
> 
> Because the strength of the Erlang model is the isolation of
> processes. Take away isolation and it's no better than Scala or Java.
> Granted, having to explicitly mark objects for sharing in D is a big
> help. Here we agree.

So we should be thinking about it, right?

>> What I discussed was a holistic approach in which language +
>> standard library provides a trusted computing base.
> 
> Have you thought about how to eliminate data races in your holistic
> approach? Will "shared" be forbidden in SafeD? Will library-based
> message-passing channels (and actors?) only accept simple value types
> and immutables?

Shared will be allowed in SafeD but it will be the responsibility of 
user code to ensure high-level race elimination. Shared will eliminate 
low-level races.

I think it's worth contemplating a scenario in which message passing is 
restricted to certain types.

> Andrei, you are the grand wizard of squeezing powerful abstractions
> out of a kludge of a language that is C++ with its ad-hoc support for
> generics. It's impressive and very useful, but it's also hermetic. By
> contrast, generic programming in D is relatively easy because of the
> right kind of support built into the language (compile-time
> interpreter).

Please no ad hominem, flattering or not.

> I trust that you could squeeze powerful multithreading abstraction
> out of D, even if the language/type system doesn't offer much of a
> support. But it will be hermetic. Prove me wrong by implementing a
> message queue using the current D2 (plus some things that are still
> in the pipeline).

I don't have the time, but I think it's worth looking into what a 
message queue implementation should look like.

>> How is this related to the discussion at hand? You want to put all
>>  concurrency support in the language. That is, you want to put
>> enough power into the language to be able to typecheck a variety of
>> concurrent programming primitives and patterns. This approach is
>> blindsided to the opportunity of defining some of these primitives
>> in the standard library, in unsafe/unportable D, yet offering safe
>> primitives to the user. In the process, the user is not hurt
>> because she still has access to the primitives. What she can't do
>> is define their own primitives in safe D. But I think that's as
>> useless a pursuit as adding keywords to C to allow one to implement
>> malloc() in safe C.
>> 
> 
> That's a bad analogy. I'm proposing the tightening of the type
> system, not the implementation of weak atomics. A better analogy
> would be adding immutable/const to the language. Except that I don't
> think const-correctness is as important as the safety of
> shared-memory concurrency.

It's a good analogy because you want to make possible the implementation 
of threading primitives in portable D. I am debating whether that is a 
worthy goal. I doubt it is.

>>> Without type-system support, though, such queues would have to be
>>> either unsafe (no guarantee that the client doesn't have
>>> unprotected aliases to messages), or restrict messages to very
>>> simple data structures (pass-by-value and maybe some very clever
>>> unique pointer/array template). The latter approach introduces
>>> huge complexity into the library, essentially making user-defined
>>> extensions impossible (unless the user's name is Andrei ;-) ).
>> Complexity will be somewhere. The problem is, you want to put much
>> of it in the language, and that will hit the language user too.
> 
> I'm being very careful not to hit the language user. You might have
> noticed that my primitive channel, the MVar, is less complex than a
> D2 implementation (it doesn't require "synchronized"). And the
> compiler will immediately tell you if you use it in an unsafe way.
> 
>> Semantic checking will always be harder on everyone than a human
>> being who sits down and implements a provably safe library in ways
>> that the compiler can't prove.
>> 
> 
> Be careful with such arguments. Somebody might use them to discredit
> immutability.

Let them discredit it and we'll see how strong their argument is.

>>> Let's not forget that right now D is _designed_ to support 
>>> shared-memory programming. Every D object has a lock, it supports
>>>  synchronized methods, the keyword "shared" is being introduced,
>>> etc. It doesn't look like D is moving away from shared memory. It
>>> looks more like it's adding some window dressing to the
>>> pre-existing mess and bidding its time. I haven't seen a
>>> comprehensive plan for D to tackle concurrency and I'm afraid
>>> that if D doesn't take a solid stance right now, it will miss the
>>> train.
>> I think we can safely ditch this argument. Walter had no idea what
>> to do about threads when he defined D, so he put whatever he saw
>> and understood in Java. He'd be the first to say that - actually,
>> he wouldn't tell me so, he _told_ me so.
>> 
> 
> I realize that. Except that the "shared" concept was added very
> recently.
> 
>> To me, adding concurrency capabilities to D is nothing like adding
>>  window dressing on top of whatever crap is there. Java and C++ are
>> in trouble, and doing what they do doesn't strike me as a good bet.
>> 
> 
> So far D has been doing exactly what Java and C++ are doing. My
> proposal goes way beyond that.

My problem is that I think it goes way beyond that straight in the wrong 
directions. It goes on and on and on about how to make deadlock-oriented 
programming less susceptible to races. I don't care about 
deadlock-oriented programming. I want to stay away from 
deadlock-oriented programming. I don't understand why I need a hecatomb 
of concepts and notions that help me continue using a programming style 
that is unrecommended.

> But if you mean D should not support
> shared-memory concurrency or give it only lip service, than you
> really have to come up with something revolutionary to take its
> place. This would obviously not make it into D2 or into your book. So
> essentially D2 would doomed in the concurrency department.

Message passing and functional style have been around for a while and 
form an increasingly compelling alternative to mutable sharing. We can 
support deadlock-oriented programming in addition to these for those who 
want it, but I don't think it's an area where we need to pay an arm and 
a leg for eliminating high-level races. I just think it's the wrong 
problem to work on.


Andrei