dmd memory usage/static lib/algorithm bug?

[H. S. Teoh]

On Wed, Aug 28, 2013 at 05:29:40PM +0200, Marek Janukowicz wrote:
> This is really a cross-domain issue, but I didn't feel like splitting
> it into separate posts would make sense.
> 
> I use DMD 2.063.2 on Linux 64-bit.
> 
> I have some code in my (non-trivial) application that basically
> corresponds to this:
> 
> import std.stdio, std.algorithm, std.array;
> 
> void main () {
>   struct Val {
>     int i;
>   }
>   Val[] arr;
>   arr ~= Val( 3 );
>   arr ~= Val( 1 );
>   arr ~= Val( 2 );
>   auto sorter = (Val a, Val b) { return a.i < b.i; };
>   writefln( "sorted: %s", arr.sort!(sorter));
> }
> 
> While this simple example works, I'm getting segfaults with
> corresponding code in thisi bigger project. Those segfaults can be
> traced down to algorithm.d line 8315 or another line (8358?) that use
> this "sorter" lambda I passed to "sort!" - suggesting it is a bad
> memory reference.

Possible causes that I can think of are:

1) You have a struct somewhere and your lambda closes over it (or one of
its members), but later on you return this struct to another scope and
invoke the lambda. But since structs can get moved around when passed
between different functions, the lambda's context pointer is invalid and
so it crashes. On simple programs this problem may be hidden because you
don't use the stack as much, so the old copy of the struct may still be
intact even though that part of the stack is technically no longer
valid, so the lambda may still *appear* to work.

2) There is a compiler bug that generates wrong code for a lambda.
There have been some such bugs before where it fails to recognize a
lambda, or fails to notice that a local variable is closed over, so it
doesn't move the local variable to the heap but leaves it on the stack,
where it gets invalidated afterwards, causing the lambda to crash.


> I tried to create a simple test case that would fail similarly, but to
> no avail. I can't make the code for my whole project available, so
> let's just say it's either some bug in DMD or something caused by my
> limited knowledge of D.

If you have a reliable way of reproducing the problem and can
encapsulate it into a shell script / batch file, you can use Vladimir
Panteleev's DustMite to automatically reduce your code to the minimum
for reproducing the bug. See:

	https://github.com/D-Programming-Language/tools/tree/master/DustMite


> Now the funny things begin: I copied algorithm.d to my project in an
> attempt to make some modifications to it (hopefully to fix the problem
> or at least get some insight into its nature), but things miraculously
> started working!  This leads me to the suspicion there is something
> wrong with libphobos2.a file provided with DMD tarball.

This is another possibility. :)  Did you check whether DMD is linking
the correct version of libphobos2.a into your program? Sometimes strange
things can happen when you have stale copies of older versions of
libphobos2.a lying around your system, and DMD accidentally picks those
up instead of the correct version.

But it could also be, that this is merely masking the problem. Invalid
pointers are notorious for causing heisenbugs that appear/disappear when
you move unrelated code around.


> Next problem in the line is that compilation of my project with
> algorithm.d included takes almost 4GB of RAM. While I'm aware of the
> fact DMD deliberately doesn't free the memory for performance
> purposes, this makes the compilation fail due to insufficient memory
> on machines with 4GB RAM (and some taken).

You could try splitting it up. :) Well, we're planning to split it up at
some point, now that DMD supports package.d. Here's roughly how you
might do it:

- Temporarily rename std/algorithm.d into another file.
- Create a directory called std/algorithm/
- Create the file std/algorithm/package.d containing something like this:

	module std.algorithm;
	public import std.algorithm.search;
	public import std.algorithm.sort;
	public import std.algorithm.set;
	public import std.algorithm.mutation;
	...

- Split up algorithm.d into the above parts (std/algorithm/search.d,
  std/algorithm/sort.d, ... etc.). You probably don't really need to
  follow exactly the above division; any partitioning of std.algorithm
  into mutually-independent parts will do. Probably splitting into just
  two parts will to cut down memory usage enough to make it compilable
  on your system.

Or, if this is too much work for your purposes, you could make a copy of
std.algorithm, rename it to my.algorithm, say, update all your imports
accordingly, and then just edit the file and delete the parts you don't
use. For example, if you don't use any of the set functions (merge,
cartesianProduct, etc.), just delete them from the file along with all
their unittests. This should reduce the amount of memory needed to
compile it.


> So my questions are:
> - how can I limit DMD memory usage?

I'll let others answer, since I'm not that familiar with DMD source code
myself.


> - how can I include a static library with my project? I can compile
> algorithm.d to a static lib, but how do I include this one explicitly
> with my project while the rest of Phobos should be taken from the
> stock libphobos2.a ?

This could be tricky. One way to do it, is to rename std.algorithm to
my.algorithm (as mentioned above), compile that into myalgo.a, and then
do something like:

	dmd program.d ... -ofprogram -L-lalgo.a -L-L.

Hope this helps.


T

-- 
Just because you survived after you did it, doesn't mean it wasn't stupid!