What is the best way to use requests and iopipe on gzipped JSON file

[Andrew Edwards]

On Friday, 13 October 2017 at 19:17:54 UTC, Steven Schveighoffer 
wrote:
> On 10/13/17 2:47 PM, Andrew Edwards wrote:
>> A bit of advice, please. I'm trying to parse a gzipped JSON 
>> file retrieved from the internet. The following naive 
>> implementation accomplishes the task:
>> 
>>      auto url = 
>> "http://api.syosetu.com/novelapi/api/?out=json&lim=500&gzip=5";
>>      getContent(url)
>>          .data
>>          .unzip
>>          .runEncoded!((input) {
>>              ubyte[] content;
>>              foreach (line; input.byLineRange!true) {
>>                  content ~= cast(ubyte[])line;
>>              }
>>              auto json = (cast(string)content).parseJSON;
>
> input is an iopipe of char, wchar, or dchar. There is no need 
> to cast it around.

In this particular case, all three types (char[], wchar[], and 
dchar[]) are being returned at different points in the loop. I 
don't know of any other way to generate a unified buffer than 
casting it to ubyte[].

> Also, there is no need to split it by line, json doesn't care.

I thought as much but my mind was not open enough to see the 
solution.

> Note also that getContent returns a complete body, but unzip 
> may not be so forgiving. But there definitely isn't a reason to 
> create your own buffer here.
>
> this should work (something like this really should be in 
> iopipe):
>
> while(input.extend(0) != 0) {} // get data until EOF

This!!! This is what I was looking for. Thank you. I incorrectly 
assumed that if I didn't process the content of input.window, it 
would be overwritten on each .extend() so my implementation was:

ubyte[] json;
while(input.extend(0) != 0) {
     json ~= input.window;
}

This didn't work because it invalidated the Unicode data so I 
ended up splitting by line instead.

Sure enough, this is trivial once one knows how to use it 
correctly, but I think it would be better to put this in the 
library as extendAll().

> And then:
> Eventually, something like this will be possible with 
> jsoniopipe (I need to update and release this too, it's 
> probably broken with some of the changes I just put into 
> iopipe). Hopefully combined with some sort of networking 
> library you could process a JSON stream without reading the 
> whole thing into memory.

That would be awesome. Again, thank you very much.

Andrew