[Issue 11862] New: Using byChunk inside a std.parallelism's parallel loop causes crash when compiled in 32 bits

[d-bugmail at puremagic.com]

https://d.puremagic.com/issues/show_bug.cgi?id=11862

           Summary: Using byChunk inside a std.parallelism's parallel loop
                    causes crash when compiled in 32 bits
           Product: D
           Version: D2
          Platform: x86_64
        OS/Version: Windows
            Status: NEW
          Severity: major
          Priority: P2
         Component: Phobos
        AssignedTo: nobody at puremagic.com
        ReportedBy: ancien9 at gmail.com


--- Comment #0 from Louis Geuten <ancien9 at gmail.com> 2014-01-03 07:23:48 PST ---
The following code produce some weird behavior when compiled in 32bits on my
Windows 7 64b machine. Removing the parallel or compiling in 64b removes the
problems.

import std.md5;
import std.stdio;
import std.file;
import std.conv;
import std.getopt;
import std.string;
import std.process;
import std.parallelism;
import std.exception;

struct Entry
{
    string name;
    ubyte[16] md5;
}

void main(string[] args)
{
    string folder1 = args[1];
    string folder2 = args[2];

    Entry[] entries1;
    foreach (name; parallel(dirEntries(folder1, SpanMode.breadth), 1)) // not
working
    {
        if(name.isFile())
        {
            entries1 ~= Entry(name ,mdFile(name));
        }
    }

    writeln(folder1," has ",entries1.length, " entries");
    Entry[] entries2;
    foreach (string name; dirEntries(folder2, SpanMode.breadth)) //working fine
    {
        if(name.isFile())
        {
            entries2 ~= Entry(name ,mdFile(name));
        }
    }
    writeln(folder2," has ", entries2.length, " entries");
}
/// Digests a file and prints the result.
ubyte[16] mdFile(string filename)
{
    MD5_CTX context;
    ubyte[16] digest;
    context.start();
    File f = File(filename,"r");

    foreach (buffer; f.byChunk(4096))
        context.update(buffer);

    context.finish(digest);
    try{
        f.close();
    }
    catch(ErrnoException e)
    {
        writeln(e.errno);
    }
    writefln("MD5 (%s) = %s", filename, digestToString(digest));
    return digest;
}

When executed, the program starts opening files and calculating their MD5.
After a short amount of files have been scanned, depending on the WorkUnitSize
(between 2 and 10 if set to 1, if default to 100, around 30-40 are processed),
the program begins malfunctioning 
- Seemingly random files throw an ErrnoException (error n° 0) when closed (not
interrupting the rest of the program)
- The program sometime just freeze (need to close it manually)
- The program generates "Object.Error: Access violation"
- The program crashes (program.exe has stopped working).

The culprit seems to be the function byChunk used to calculate the MD5.
Using std.stream like suggested in this question's answer (
http://stackoverflow.com/questions/20888596/opening-files-in-a-parallel-loop-crash-the-program
) makes the program work in 32 and 64bits.

While this issue can be fixed by compiling in 64bit, it's still a major
problem.

PS : It's my first time reporting a bug, so if some information is lacking or
I've submitted this wrong, please let me know and I'll try to fix it.

-- 
Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------