What is the memory usage of my app?
via Digitalmars-d-learn
digitalmars-d-learn at puremagic.com
Thu Apr 16 13:33:16 PDT 2015
On Thursday, 16 April 2015 at 17:13:25 UTC, Laeeth Isharc wrote:
> For CSV files, what I found is that parsing is quite slow (and
> memory intensive).
If your sure that CSV reading is the culprit, writing a custom
parser could help. It's possible to load a CSV file with almost
no memory overhead. What I would do:
- Use std.mmfile with Mode.readCopyOnWrite to map the file into
memory.
- Iterate over the lines, and then over the fields using
std.algorithm.splitter.
- Don't copy, but return slices into the mapped memory.
- If a field needs to be unescaped, this can be done in-place.
Unescaping never makes a string longer, and the original file
won't be modified thanks to COW (private mapping).
More information about the Digitalmars-d-learn
mailing list