Help optimizing UnCompress for gzipped files

[Christian Köstlin]

On 05.01.18 23:04, Steven Schveighoffer wrote:
> On 1/5/18 3:09 PM, Christian Köstlin wrote:
>> On 05.01.18 15:39, Steven Schveighoffer wrote:
>>> Yeah, I guess most of the bottlenecks are inside libz, or the memory
>>> allocator. There isn't much optimization to be done in the main program
>>> itself.
>>>
>>> D compiles just the same as C. So theoretically you should be able to
>>> get the same performance with a ported version of your C code. It's
>>> worth a shot.
>> I added another version that tries to do the "same" as the c version
>> using mallocator, but i am still way off, perhaps its creating too many
>> ranges on the underlying array. but its around the same speed as your
>> great iopipe thing.
> 
> Hm... I think really there is some magic initial state of the allocator,
> and that's what allows it to go so fast.
> 
> One thing about the D version, because druntime is also using malloc
> (the GC is backed by malloc'd data after all), the initial state of the
> heap is quite different from when you start in C. It may be impossible
> or nearly impossible to duplicate the performance. But the flipside (if
> this is indeed the case) is that you won't see the same performance in a
> real-world app anyway, even in C.
> 
> One thing to try, you preallocate the ENTIRE buffer. This only works if
> you know how many bytes it will decompress to (not always possible), but
> it will take the allocator out of the equation completely. And it's
> probably going to be the most efficient method (you aren't leaving
> behind smaller unused blocks when you realloc). If for some reason we
> can't beat/tie the C version doing that, then something else is going on.
yes ... this is something i forgot to try out ... will do now :)
mhh .. interesting numbers ... c is even faster, my d lowlevel solution
is also a little bit faster, but much slower than the no copy version
(funnily, no copy is the wrong name, it just overwrites all the data in
a small buffer).

>> My solution does have the same memory leak, as I am not sure how to best
>> get the memory out of the FastAppender so that it is automagically
>> cleaned up. Perhaps if we get rc things, this gets easier?
> 
> I've been giving some thought to this. I think iopipe needs some buffer
> management primitives that allow you to finagle the buffer. I've been
> needing this for some time anyway (for file seeking). Right now, the
> buffer itself is buried in the chain, so it's hard to get at the actual
> buffer.
> 
> Alternatively, I probably also need to give some thought to a mechanism
> that auto-frees the memory when it can tell nobody is still using the
> iopipe. Given that iopipe's signature feature is direct buffer access,
> this would mean anything that uses such a feature would have to be unsafe.
yes .. thats tricky ...
one question about iopipe. is it possible to transform the elements in
the pipe as well ... e.g. away from a buffer of bytes to json objects?

--
Christian Köstlin