Splitting up large dirty file

[Dennis]

On Wednesday, 16 May 2018 at 08:20:06 UTC, drug wrote:
> What is the purpose of `.drop(4)`? I'm pretty sure this is the 
> reason of the exception.

The file in question is a .json database dump with an array 
"rows" of 10 million 8-line objects. The newlines in the string 
fields are escaped, but they still contain other invalid 
characters which makes std.json reject it.

The first 4 lines of the file are basically "header" and the last 
2 lines are a closing ] and }, so I want to split every 4 + 
8*(10_000_000/amountOfFiles) n lines and also remove trailing the 
comma, add brackets, drop the last 2 lines etc.

I thought it wouldn't be hard to crudely split this file using 
D's range functions and basic string manipulation, but the 
combination of being to large for a string and having invalid 
encoding seems to defeat most simple solutions. For now I decided 
to use Git Bash and do:
tail -n80000002 inputfile.json | split -l 8000000 - outputfile

And now I have files that do fit in memory. I'm still interested 
in complete D solutions though, thanks for the iopipe and memory 
mapped file suggestions Steven and Jonathan. I will check those 
out.