DMD 1.029 and 2.013 releases

[Sean Kelly]

== Quote from Walter Bright (newshound1 at digitalmars.com)'s article
> Sean Kelly wrote:
> >>> Look at shared-
> >>> memory multiprogramming for instance.  It's quite easy and understandable
> >>> to share a few data structures between threads
> >> It is until one of those threads tries to change the data.
> >
> > I suppose I should have been more clear.  An underlying assumption of mine is
> > that no thread maintains references into shared data unless they hold the lock
> > that protects that data.
> The problem with locks are:
> 1) they are expensive, so people try to optimize them away (grep for
> "double checked locking")
> 2) people forget to use the locks
> 3) deadlocks

1) The cost of acquiring or committing a lock is generally roughly equivalent to
     a memory synchronization, and sometimes less than that (futexes, etc).  So
     it's not insignificant, but also not as bad as people seem to think.  I suspect
     that locked operations are often subject to premature optimization.
2) If locking is built into the API then they can't forget.
3) Deadlocks aren't typically an issue with the approach I described above because
     it largely eliminates the chance that the programmer will call into unknown code
     while holding a lock.

I do think that locks stink as a general multiprogramming tool, but they can be
quite useful in implementing more complex multiprogramming tools, if nothing
else.  Also, they can be about the fastest option in some cases, and this can be
important.  For example, locks are much faster than transactional memory--they
just introduce problems like priority inversion and deadlock (fun fun).  That said,
transactional memory can result in livelock, so neither is a clear win.

> >>> (which I'd assert is the original
> >>> intent anyway), but common practice among non-experts is to use mutexes
> >>> to protect code rather than data, and to call across threads willy-nilly.  It's
> >>> no wonder the commonly held belief is that multiprogramming is hard.
> >> The "multiprogramming is hard" is not based on a misunderstanding. It
> >> really is hard.
> >
> > My claim is that multiprogramming is hard because the ability to share memory
> > has been mis-used.  It's not hard in general, in my opinion.
> When people as smart and savvy as Scott Meyers find it confusing, it's
> confusing. (Scott Meyers wrote the definitive paper on doubled checked
> locking, and what's wrong with it.) Heck, I have a hard enough time
> explaining what the difference between const and invariant is, how is
> memory coherency going to go down? <g>

Fair enough :-)

> >>> Regarding lock-free programming in particular, I think it's worth pointing
> >>> out that leaving out support for lock-free programming in general excludes
> >>> an entire realm of code being written--not only library code to be ultimately
> >>> used by everyday programmers, but kernel code and such as well.  Look at
> >>> the Linux source code, for example.
> >> I agree that lock free programming is important, but volatile doesn't
> >> get you there.
> >
> > How is it lacking?  I grant that it's very low-level, but it does address the key
> > concern for lock-free programming.
> volatile actually puts locks around accesses (at least in the Java
> memory model it does). So, it doesn't get you lock-free programming.
> Just avoiding caching of reloads is not the key to lock-free
> programming. There's the ordering problem.

I must be missing something... I thought 'volatile' addressed compiler reordering as
well?  That aside, I do think that the implementation of 'volatile' in D 1.0 is too
complicated for the average programmer to use correctly and thus may not be the
perfect solution for D, but I also think that it solves the language/compiler part of
the problem.

> >>  >  As for the C++0x discussions, I feel
> >>  > that some of the participants of the memory model discussion are experts
> >>  > in the field and understand quite well the issues involved.
> >> Yes, there are a handful who do really understand it (Hans Boehm and
> >> Herb Sutter come to mind). If only the rest of us were half as smart <g>.
> >
> > My personal belief is that the issue is really more a lack of plain old explanation
> > of the concepts than anything else.  The topic is rarely discussed outside of
> > research papers, and most other documentation is either confusing or just plain
> > wrong (the IA-86 memory model spec comes to mind, for example).  Not to
> > belittle the knowledge or experience of the C++ folks in any respect--this is
> > simply my experience with the information surrounding the topic :-)
> I've seen many attempts at explaining it, including presentations by
> Herb Sutter himself. Sorry, but most of the audience doesn't get it.
> I attended a conference a couple years back on what do do about adding
> multithreading support to C++. There were about 30 attendees, pretty
> much the top guys in C++ programming, including Herb Sutter and Hans
> Boehm. Herb and Hans did most of the talking, and the rest of us sat
> there wondering "what's a cubit". Things have improved a bit since then,
> but it's pretty clear that the bulk of programmers are never going to
> get it, and getting mp programs to work will have the status of a black art.
> What's needed is something like what garbage collection did for memory
> management. The language has to take care of synchronization
> *automatically*. Being D, of course there will be a way for the
> sorcerers to practice the black art, but for the rest of us there needs
> to be a reliable and reasonable alternative.

I very much agree.  My real interest in preserving the black arts in D is so that
library developers can produce code which solves these problems in a more
elegant manner, whatever that may be.  I don't have any expectation that the
average programmer would ever want or need to use something like 'volatile'
or even ordered atomics.  It's far too low-level of a solution to the problem at
hand.  However, if this can be accomplished without any language facilities at
all then I'm all for it.  I simply don't want to have to rely on compiler-specific
knowledge when writing code, be it at a high level or a low level.


Sean