What is the best way to use requests and iopipe on gzipped JSON file
Steven Schveighoffer
schveiguy at yahoo.com
Fri Oct 13 22:22:54 UTC 2017
On 10/13/17 6:18 PM, ikod wrote:
> On Friday, 13 October 2017 at 19:17:54 UTC, Steven Schveighoffer wrote:
>>
>> Eventually, something like this will be possible with jsoniopipe (I
>> need to update and release this too, it's probably broken with some of
>> the changes I just put into iopipe). Hopefully combined with some sort
>> of networking library you could process a JSON stream without reading
>> the whole thing into memory.
>
> This can be done with requests. You can ask not to load whole content in
> memory, but instead produce input range, which will continue to load
> data from server when you will be ready to consume:
>
> auto rq = Request();
> rq.useStreaming = true;
> auto rs = rq.get("http://httpbin.org/image/jpeg");
> auto stream = rs.receiveAsRange();
> while(!stream.empty) {
> // stream.front contain next data portion
> writefln("Received %d bytes, total received %d from document
> legth %d", stream.front.length, rq.contentReceived, rq.contentLength);
> stream.popFront; // continue to load from server
> }
Very nice, I will add a component to iopipe that converts a "chunk-like"
range like this into an iopipe source, as this is going to be needed to
interface with existing libraries. I still will want to skip the middle
man buffer at some point though :)
Thanks!
-Steve
More information about the Digitalmars-d-learn
mailing list