Java > Scala

[Russel Winder]

On Sun, 2011-12-18 at 17:08 -0600, Andrei Alexandrescu wrote:
> On 12/18/11 1:08 PM, Isaac Gouy wrote:
> >> From: Russel Winder
> >> Subject: Re: Java>  Scala
> >> Newsgroups: gmane.comp.lang.d.general
> >> Sat, 17 Dec 2011 22:18:26 -0800
> >
> >> I really rather object to being labelled an educated idiot.
> > ....
> >> If you want to look at even more biased benchmarking look at
> >> http://shootout.alioth.debian.org/ it is fundamentally designed to show
> >> that C is the one true language for writing performance computation.
> >
> > I rather object to the baseless accusation that the benchmarks game is "designed to show that C is the one true language for writing performance computation."

Overstated perhaps, baseless, no.  But this is a complex issue.

> > Your accusation is false.
> >
> > Your accusation is ignorant (literally).

The recent thread between Caligo, myself and others on this list should
surely have displayed the futility of arguing in this form.

> It also strikes me as something rather random to say. Far as I can tell 
> the shootout comes with plenty of warnings and qualifications and uses a 
> variety of tests that don't seem chosen to favor C or generally systems 
> programming languages.

The Shootout infrastructure and overall management is great.  Isaac has
done a splendid job there.  The data serves a purpose for people who
read between the lines and interpret the results with intelligence.  The
opening page does indeed set out that you have to be very careful with
the data to avoid comparing apples and oranges.  The data is presented
in good faith.

The system as set out is biased though, systematically so.  This is not
a problem per se since all the micro-benchmarks are about
computationally intensive activity.  Native code versions are therefore
always going to appear better.  But then this is fine the Shootout is
about computationally intensive comparison.  Actually I am surprised
that Java does so well in this comparison due to its start-up time
issues.

Part of the "problem" I alluded to was people using the numbers without
thinking.  No amount of words on pages affect these people, they take
the numbers as is and make decisions based solely on them.  C, C++ and
Fortran win on most of them and so are the only choice of language.  (OK
so Haskell wins on the quad-core thread-ring, which I find very
interesting.)

As I understand it, Isaac ruins this basically single handed, relying of
folk providing versions of the code.  This means there is a highly
restricted resource issue in managing the Shootout.  Hence a definite
set of problems and a restricted set of languages to make management
feasible.  This leads to interesting situation such as D is not part of
the set but Clean and Mozart/Oz are.  But then Isaac is the final
arbiter here, as it is his project, and what he says goes.

I looked at the Java code and the Groovy code a couple of years back (I
haven't re-checked the Java code recently), and it was more or less a
transliteration of the C code.  This meant that the programming
languages were not being shown off at their best.  I started a project
with the Groovy community to provide reasonable version of Groovy codes
and was getting some take up.  Groovy was always going to be with Python
and Ruby and nowhere near C, C++, and Fortran, or Java, but the results
being displayed at the time were orders of magnitude slower than Groovy
could be, as shown by the Java results.  The most obvious problem was
that the original Groovy code was written so as to avoid any parallelism
at all.

Of course Groovy (like Python) would never be used directly for this
sort of computation, a mixed Groovy/Java or Python/C (or Python/C++,
Python/Fortran) would be -- the "tight loop" being coded in the static
language, the rest in the dynamic language.   Isaac said though that
this was not permitted, that only pure single language versions were
allowed.  Entirely reasonable in one sense, unfair in another: fair
because it is about language performance in the abstract, unfair because
it is comparing languages out of real world use context.

(It is worth noting that the Python is represented by CPython, and I
suspect PyPy would be a lot faster for these micro-benchmarks.  But only
when PyPy is Python 3 compliant since Python 3 and not Python 2 is the
representative in the Shootout.  A comparison here is between using
Erlang and Erlang HiPE.)

In the event, Isaac took Groovy out of the Shootout, so the Groovy
rewrite effort was disbanded.  I know Isaac says run your own site, but
that rather misses the point, and leads directly to the sort of hassles
Walter had when providing a benchmark site.   There is no point in a
language development team running a benchmark.  The issues are
perceived, if not real, bias in the numbers.  Benchmarks have to be run
by an independent even if the contributions are from language
development teams.

> But I'm sure Russel had something in mind. Russel, would you want to 
> expand a bit?

Hopefully the above does what you ask.

The summary is that Isaac is running this in good faith, but there are
systematic biases in the whole thing, which is entirely fine as long as
you appreciate that.

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puremagic.com/pipermail/digitalmars-d/attachments/20111220/04998f97/attachment-0001.pgp>