Splitting up large dirty file

[Jonathan M Davis]

On Tuesday, May 15, 2018 20:36:21 Dennis via Digitalmars-d-learn wrote:
> I have a file with two problems:
> - It's too big to fit in memory (apparently, I thought 1.5 Gb
> would fit but I get an out of memory error when using
> std.file.read)
> - It is dirty (contains invalid Unicode characters, null bytes in
> the middle of lines)
>
> I want to write a program that splits it up into multiple files,
> with the splits happening every n lines. I keep encountering
> roadblocks though:
>
> - You can't give Yes.useReplacementChar to `byLine` and `byLine`
> (or `readln`) throws an Exception upon encountering an invalid
> character.
> - decodeFront doesn't work on inputRanges like
> `byChunk(4096).joiner`
> - std.algorithm.splitter doesn't work on inputRanges either
> - When you convert chunks to arrays, you have the risk of a split
> being in the middle of a character with multiple code units
>
> Is there a simple way to do this?

If you're on a *nix systime, and you're simply looking for a solution to
split files and don't necessarily care about writing one, I'd suggest trying
the split utility:

https://linux.die.net/man/1/split

If I had to write it in D, I'd probably just use std.mmap and operate on the
files as a dynamic array of ubytes, since if what you care about is '\n',
that can easily be searched for without needing any decoding, and using mmap
avoids having to chunk anything.

- Jonathan M Davis