What is the best way to use requests and iopipe on gzipped JSON file

[Steven Schveighoffer]

On 10/13/17 6:18 PM, ikod wrote:
> On Friday, 13 October 2017 at 19:17:54 UTC, Steven Schveighoffer wrote:
>>
>> Eventually, something like this will be possible with jsoniopipe (I 
>> need to update and release this too, it's probably broken with some of 
>> the changes I just put into iopipe). Hopefully combined with some sort 
>> of networking library you could process a JSON stream without reading 
>> the whole thing into memory.
> 
> This can be done with requests. You can ask not to load whole content in 
> memory, but instead produce input range, which will continue to load 
> data from server when you will  be ready to consume:
> 
>      auto rq = Request();
>      rq.useStreaming = true;
>      auto rs = rq.get("http://httpbin.org/image/jpeg");
>      auto stream = rs.receiveAsRange();
>      while(!stream.empty) {
>          // stream.front contain next data portion
>          writefln("Received %d bytes, total received %d from document 
> legth %d", stream.front.length, rq.contentReceived, rq.contentLength);
>          stream.popFront; // continue to load from server
>      }

Very nice, I will add a component to iopipe that converts a "chunk-like" 
range like this into an iopipe source, as this is going to be needed to 
interface with existing libraries. I still will want to skip the middle 
man buffer at some point though :)

Thanks!

-Steve