What to do with InvalidMemoryOperationError

[anonymous via Digitalmars-d-learn]

On Wednesday, 21 January 2015 at 13:07:11 UTC, Nordlöw wrote:
> On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
>> On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
>>> My executable throws as
>>>
>>>   core.exception.InvalidMemoryOperationError@(0)
>>
>> I've tracked it down to being caused by
>>
>>    foreach (line; File(path).byLine) {}
>>
>> when path contains a very large text file (392 MB, 1658080 
>> lines).
>>
>> Do I have any alternatives to this that doesn't trigger 
>> exception?
>
> I get the exact same behaviour when I try using MmFile instead:
>
>     import std.mmfile: MmFile;
>     auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, 
> pageSize);
>     const data = cast(char[])mmf[];
>     foreach (line; data.splitter('\n') {}
>
> Does byLine use splitter? If so, maybe splitter is the 
> problem...

I tested it with a generated large file, and the (rough) file 
size and line count are apparently not enough to trigger the bug.

Could you share the text file? Maybe it can be compressed to a 
manageable size? Or maybe you can reduce it to a manageable size?

A 'binary reduction' might work: Take the first half of the file, 
and test with that. Error? Proceed with that half. No error? Try 
the other half. When both halves produce no error, give up.

Or maybe dustmite can help here?

If the file contains sensitive information, and you cannot reduce 
it to a reasonable size, you may be able to programmatically 
replace classes of characters with one character. E.g. replace 
all alphanumeric characters with 'x'. Be cautious of replacing 
multibyte characters with single bytes. And newlines should 
probably be left intact.