Parallel Rogue-like benchmark

[H. S. Teoh]

On Fri, Aug 23, 2013 at 08:25:20PM -0700, Walter Bright wrote:
> On 8/23/2013 7:10 PM, Jesse Phillips wrote:
> >If we decided that 2 lines was how we do formatting,
> 
> In general, I regard a "line of code" as one statement or one
> declaration. Comments don't count, nor does cramming 3 statements into
> one line make it one LOC.
> 
> Of course, you can still game that, but if you're reasonable then it
> is a reasonable measure.

I am still skeptical of LOC as a reliable measure of language
complexity. How many LOC does the following code have?

	auto formatYear(int year, int monthsPerRow)
	{
	    enum colSpacing = 1;
	    return
		datesInYear(year)
		.byMonth()
		.chunks(monthsPerRow)
		.map!(r =>
			r.formatMonths()
			 .array()
			 .pasteBlocks(colSpacing)
			 .join("\n"))
		.join("\n\n");
	}

It's 14 lines using the above formatting, but the { and } are arguably
discountable, which makes it 12 lines of code.

But by your metric of one statement / one declaration, this is only 3
lines of code. Or, if you count the lambda separately, 4 lines of code.

That's a variation by a factor of at least 3, which makes it an
unreliable metric.

So what would you do? Declare that each function call should count as 1
LOC? How then would you count the LOC in the following code?

	void main(string[] args) {
	    int x = max(2, min(5, args[1].to!int));
	}

Is it 2 lines of code or 4 (since it's 3 function calls + 1 declaration
of main)? Again, you have a variation by a factor of 2, which makes any
results derived from LOC unreliable at best.

In none of the above examples did I try to deliberately game with the
metric. But the metric is still pretty inaccurate, and requires
subjective judgment calls.

A far more reliable measure of code complexity is to look at the
compressed size of the source code (e.g., with zip), which is an
approximation of the Kolgomorov complexity of the text, roughly
equivalent to the amount of information content it has.

Say you're comparing two functionally equivalent programs P1 and P2 in
two languages L1 and L2, respectively. Then the compressed size of P1
and P2, respectively, measure roughly how complex the source code must
be in order to express the program. The smaller the compressed size, the
more expressive the language, because it means that you only need to use
a small amount of complexity in order to express the program.

Also interesting is the ratio of the uncompressed source code size to
its compressed size: this ratio tells you how verbose the language is.
The higher the ratio, the more verbose the language: roughly speaking,
it requires more characters to represent a given amount of complexity.
If this ratio is low, it means that the source code is pretty terse; it
only requires a small amount of characters to express a given amount of
complexity.

While the above is obviously still an approximate metric, it at least
has some scientific basis in information theory, and I would put more
trust in this than in LOC. Already, it tells you that there are two
related but different factors at play: the inherent expressiveness of
the language (the absolute size of the compressed source code of a
program of fixed functionality, given a fixed compression algorithm),
and the verbosity of its syntax (the ratio of the uncompressed source
with the compressed source). It's the inherent expressiveness that's a
better measure of a language's expressiveness, because it doesn't
unfairly penalize a language for having, say, longer, more descriptive
identifiers, but it does penalize a language for requiring more
convoluted structures to express a given task.  Verbosity may matter
more to us human coders, especially those of us who like text editors,
but it doesn't fairly measure the inherent expressiveness of the
language.

LOC not only doesn't even begin to approximate Kolgomorov complexity, it
conflates these two related but ultimately different characteristics of
the language. Thus, any conclusions drawn from LOC are bound to be
highly unreliable.


T

--
Designer clothes: how to cover less by paying more.