What is the best way to use requests and iopipe on gzipped JSON file
ikod
geller.garry at gmail.com
Tue Oct 17 08:33:49 UTC 2017
Hello, Steve
On Friday, 13 October 2017 at 22:22:54 UTC, Steven Schveighoffer
wrote:
> On 10/13/17 6:18 PM, ikod wrote:
>> On Friday, 13 October 2017 at 19:17:54 UTC, Steven
>> Schveighoffer wrote:
>>>
>>> Eventually, something like this will be possible with
>>> jsoniopipe (I need to update and release this too, it's
>>> probably broken with some of the changes I just put into
>>> iopipe). Hopefully combined with some sort of networking
>>> library you could process a JSON stream without reading the
>>> whole thing into memory.
>>
>> This can be done with requests. You can ask not to load whole
>> content in memory, but instead produce input range, which will
>> continue to load data from server when you will be ready to
>> consume:
>>
>> auto rq = Request();
>> rq.useStreaming = true;
>> auto rs = rq.get("http://httpbin.org/image/jpeg");
>> auto stream = rs.receiveAsRange();
>> while(!stream.empty) {
>> // stream.front contain next data portion
>> writefln("Received %d bytes, total received %d from
>> document legth %d", stream.front.length, rq.contentReceived,
>> rq.contentLength);
>> stream.popFront; // continue to load from server
>> }
>
> Very nice, I will add a component to iopipe that converts a
> "chunk-like" range like this into an iopipe source, as this is
> going to be needed to interface with existing libraries. I
> still will want to skip the middle man buffer at some point
> though :)
>
> Thanks!
>
> -Steve
Just in order to have complete picture here - getContent returns
not just ubyte[], but more rich structure (which can be convered
to ubyte[] if needed). Basically it is an
immutable(immutable(ubyte)[]) and almost all data there are just
data received from network without any data copy.
There are more details and docs on
https://github.com/ikod/nbuff/blob/master/source/nbuff/buffer.d.
Main goal behind Buffer is to minimize data movement, but it also
support many range properties, as long as some internal optimized
methods.
Thanks,
Igor
More information about the Digitalmars-d-learn
mailing list