Revamped concurrency API

[Jeremie Pelletier]

bearophile wrote:
> Bartosz Milewski:
> 
>> It's good to know that my ideas are still circulating in the D community. Thanks, guys!<
> 
> They will keep circulating! And there's not just D in computer science, future computer languages may use your ideas. Someday I'll design my own C-like language, simpler (and more limited in scope) than D ;-)
> 
> 
>> Since I'm now put on the spot, I have to explain my recent detachment from D.<
> 
> This makes me (and surely many other persons here) a little sad. 

Same here, I read your blog with much interest and check it almost daily 
for updates, your ideas about concurrency are really inspiring!

I'm also sure many of us here, me included, did our first steps in GUI 
programming years ago by following your tutorial on reliable software.

> D is an engineering project, and it's not small. And the first thing to learn about all sizeable real-world engineering projects (like the creation of a large bridge, or many other things) is that they are first of all politics. They are not the result of rationality, even if often some rationality is required. All large engineering project come out of a large amount of compromises. If you pretend to see one of such systems to come out of rationality alone, you will never see a real one. 
> 
> So every time you have to fight for the rationality to prevail, and you have to be ready to accept some compromises. It's life.

I still have faith that Walter and Andrei can filter out all the ideas 
here no matter how heated up the arguments and come up with a design 
that will be a decent compromise in the requested features and design 
changes.

>> it's hard for me to subscribe to the "good enough" philosophy (as long as it's better that C++, it's fine for D).<
> 
> D language is largely grown, and not designed. D is not Scala or Haskell. I think there is not enough brain mass in this group to design rationally a C++-class language. Probably there's not even brain mass to design a Python-class language. D2 will surely have a big amount of design mistakes, every one here can list you some of them. But one of the best qualities of Walter is that he's not frozen, he sometimes seems to move very slowly, but he never stops moving, and usually he eventually sees the light (and when this happens it's usually a small surprise for everyone else, like the show of a rabbit pulled out of a hat). So I think some of the design mistakes of D2 will be fixed in D3 :-)

The brain mass is there, more than enough too. Its the manpower that 
lacks, we propose way more features than we have manpower to implement 
in time to either release them or properly test them and see if the 
design works as good as wanted.

A lot of us would like to contribute to the compiler and library but 
we're already busy with our daily jobs, maybe Walter needs to hire half 
this newsgroup and build us an office :o)

>> My impression is that, as the release of D2 and the publication of Andrei's book are nearing, this attitude is gaining ground. I try to fight this attitude but it's an uphill battle. Or, as Andrei puts it, I start whining and give up ;-).<
> 
> Andrei's book comes too much early, forcing rhythms that I think aren't good for D2. On the other hand no software is ever finished, it's just abandoned. And you must not think of D2 as the finished product :-) You may even think of D2 as just a bridge toward D3 :-) D2 is probably just an experiment, to experimentally see what design ideas are good and what to throw away and redesign. And such things are slow. So you may want to come back in few years, to try to improve the concurrency design of D3 :-)

I agree here, while I think Andrei's book is a must for the community to 
grow, it should rush the compiler. I strongly believe D2 needs proper 
shared and lent semantics, or else we're just creating a stepping stone 
for D3, leaving D2 in a state that isn't suited for concurrent model 
implementations in a type-safe manner.

I don't mind however getting the spec ready with the features coming in 
the near future as part of D2 (either in patch releases or minor 
releases, ie 2.1).

>> The semantics of "shared." I can live with postponing the implementation of the race-free type system, but not with the compiler inserting barriers around all shared reads and writes, even inside synchronized sections.<
> 
> This is a topic that can be discussed. If enough people think this is a problem, then eventually Walter may change his mind, or he may design something better, etc. If no one discussed problems they can't be fixed/improved. I know that discussing things and accepting compromises is a pain, but it's an inevitable pain. Rationality alone usually never wins, you need politics too.

I brought up this topic a number of time without much success. Shared is 
hard to work with right now without tons of casting. It's also way too 
easy to accidentally write recursive locks with the current synchronized 
model, which is a nice fallback for the code to run, but generates way 
more calls than is actually needed.

If what we need are politics, then the republic of J (aka my ego) 
declares we need shared and lent qualifier in D that are as easy to use 
without a casting mess as immutable and const are, the rest of the 
concurrent details will follow in the many, many library implemented models.

>> The C++-derived template metaprogramming mess. Especially when C++0x provides better support for variadic templates than D (see my upcoming blog). I fought successfully for non-functional approach to string mixins. The same is needed for more general templates. In my opinion, there should not be any part of the language that is understandable only by seasoned library writers.<
> 
> Again, this seems a small piece of D2, so it may be fixed if enough people think this is bad. You can explain what do you want to change here, and people will see and think. Eventually it may even be fixed.

I can't wait to read the article, I agree that forcing templates to work 
in a functional way is often counter-intuitive, took me a while to get 
used to it.

>> The "better than Java" aspect of D ignores the latest development in Java. In particular the work on non-null pointers<
> 
> We have discussed weeks about nonnull references in D (and I have even discussed about nonnull pointers too), I have done my best. A good percentage of people have said that they like to try this idea. Walter in the end was not interested (I think). But maybe in future the situation will change. There's always hope on this. At the end I have seen no one say "This is a totally wrong idea and will never see the light in D".

We just haven't found an elegant and practical solution to it yet.

>> and the use of annotations for type-system extensions. Annotations are being added to D as we speak because it finally became obvious that no amount of cleverness can make object properties work without additional syntax. So the patching begins, without a clear vision of the role of annotation in future D.<
> 
> Annotations in D are just an embryo, and I think they come from no design yet, they are just a bit copied from Java. If you have a vision for their role in D, then it's a very good moment to talk and show such vision. I am sure we will be interested to listen. It's surely not too much late to design them well.
> 
> 
>> As far as my thread work went, I had an almost working implementation of spawn, except for a nasty compiler bug which resisted all efforts to reduce it to a simple test case. Threads also required some clever work with templates (testing arguments to "spawn" for sharing violations).<
> 
> Walter isn't magic. If some feature is very hard to implement then it may be better to not add it until a good implementation is actually created.

>> I implemented typelist.d in Phobos to make it easier, and learned how inflexible D templates were. At least they are better than C++, or are they?<
> 
> D templates are not "better" than C++ ones, they are simpler to use, less bug-prone, nicer-looking, but I think they are also less flexible. In D there are many things that are less flexible than similar things in C++ (see for example operator overloading too), this can be a good thing, a design choice, because too much flexibility makes a design bug-prone, complex to implement and harder to use. So you often want less flexibility, even if in hard situations (like your one) such limits may look bad.

I partly disagree here, more flexibility is a great thing for a systems 
language, but it doesn't have to be the only way of doing things. Less 
flexibility is often only a good thing in higher level languages.

> Bye and thank you for all your work so far,
> bearophile