What is the best way to use requests and iopipe on gzipped JSON file

ikod geller.garry at gmail.com
Tue Oct 17 08:33:49 UTC 2017


Hello, Steve

On Friday, 13 October 2017 at 22:22:54 UTC, Steven Schveighoffer 
wrote:
> On 10/13/17 6:18 PM, ikod wrote:
>> On Friday, 13 October 2017 at 19:17:54 UTC, Steven 
>> Schveighoffer wrote:
>>>
>>> Eventually, something like this will be possible with 
>>> jsoniopipe (I need to update and release this too, it's 
>>> probably broken with some of the changes I just put into 
>>> iopipe). Hopefully combined with some sort of networking 
>>> library you could process a JSON stream without reading the 
>>> whole thing into memory.
>> 
>> This can be done with requests. You can ask not to load whole 
>> content in memory, but instead produce input range, which will 
>> continue to load data from server when you will  be ready to 
>> consume:
>> 
>>      auto rq = Request();
>>      rq.useStreaming = true;
>>      auto rs = rq.get("http://httpbin.org/image/jpeg");
>>      auto stream = rs.receiveAsRange();
>>      while(!stream.empty) {
>>          // stream.front contain next data portion
>>          writefln("Received %d bytes, total received %d from 
>> document legth %d", stream.front.length, rq.contentReceived, 
>> rq.contentLength);
>>          stream.popFront; // continue to load from server
>>      }
>
> Very nice, I will add a component to iopipe that converts a 
> "chunk-like" range like this into an iopipe source, as this is 
> going to be needed to interface with existing libraries. I 
> still will want to skip the middle man buffer at some point 
> though :)
>
> Thanks!
>
> -Steve

Just in order to have complete picture here - getContent returns 
not just ubyte[], but more rich structure (which can be convered 
to ubyte[] if needed). Basically it is an 
immutable(immutable(ubyte)[]) and almost all data there are just 
data received from network without any data copy.

There are more details and docs on 
https://github.com/ikod/nbuff/blob/master/source/nbuff/buffer.d. 
Main goal behind Buffer is to minimize data movement, but it also 
support many range properties, as long as some internal optimized 
methods.

Thanks,

Igor


More information about the Digitalmars-d-learn mailing list