Help optimizing UnCompress for gzipped files

[Christian Köstlin]

On 07.01.18 14:44, Steven Schveighoffer wrote:
> Not from what I'm reading, the C solution is about the same (257 vs.
> 261). Not sure if you have averaged these numbers, especially on a real
> computer that might be doing other things.
yes you are right ... for proper benchmarking proper statistics should
be in place, taking out extreme values, averaging them, ...

> Note: I would expect it to be a tiny bit faster, but not monumentally
> faster. From my testing with the reallocation, it only reallocates a
> large quantity of data once.
> 
> However, the D solution should be much faster. Part of the issue is that
> you still aren't low-level enough :)
> 
> Instead of allocating the ubyte array with this line:
> 
> ubyte[] buffer = new ubyte[200*1024*1024];
> 
> Try this instead:
> 
> // from std.array
> auto buffer = uninitializedArray!(ubyte[], 200*1024*1024);
thanks for that ... i just did not know how to get an uninitialized
array. i was aware, that dlang is nice and puts init there :)

> Yes! I am working on doing just that, but haven't had a chance to update
> the toy project I wrote: https://github.com/schveiguy/jsoniopipe
> 
> I was planning actually on having an iopipe of JsonItem, which would
> work just like a normal buffer, but reference the ubyte buffer underneath.
> 
> Eventually, the final product should have a range of JsonValue, which
> you would recurse into in order to parse its children. All of it will be
> lazy, and stream-based, so you don't have to load the whole file if it's
> huge.
> 
> Note, you can't have an iopipe of JsonValue, because it's a recursive
> format. JsonItems are just individual defined tokens, so they can be
> linear.
sounds really good. i played around with
https://github.com/mleise/fast/blob/master/source/fast/json.d ... thats
an interesting pull parser with the wrong licence unfortunately ... i
wonder if something like this could be done on top of iopipe instead of
a "real" buffer.

---
Christian Köstlin