What is the best way to use requests and iopipe on gzipped JSON file
Steven Schveighoffer
schveiguy at yahoo.com
Fri Oct 13 19:17:54 UTC 2017
On 10/13/17 2:47 PM, Andrew Edwards wrote:
> A bit of advice, please. I'm trying to parse a gzipped JSON file
> retrieved from the internet. The following naive implementation
> accomplishes the task:
>
> auto url =
> "http://api.syosetu.com/novelapi/api/?out=json&lim=500&gzip=5";
> getContent(url)
> .data
> .unzip
> .runEncoded!((input) {
> ubyte[] content;
> foreach (line; input.byLineRange!true) {
> content ~= cast(ubyte[])line;
> }
> auto json = (cast(string)content).parseJSON;
input is an iopipe of char, wchar, or dchar. There is no need to cast it
around.
Also, there is no need to split it by line, json doesn't care.
Note also that getContent returns a complete body, but unzip may not be
so forgiving. But there definitely isn't a reason to create your own
buffer here.
this should work (something like this really should be in iopipe):
while(input.extend(0) != 0) {} // get data until EOF
And then:
auto json = input.window.parseJSON;
> foreach (size_t ndx, record; json) {
> if (ndx == 0) continue;
> auto title = json[ndx]["title"].str;
> auto author = json[ndx]["writer"].str;
> writefln("title: %s", title);
> writefln("author: %s\n", author);
> }
> });
>
> However, I'm sure there is a much better way to accomplish this. Is
> there any way to accomplish something akin to:
>
> auto url =
> "http://api.syosetu.com/novelapi/api/?out=json&lim=500&gzip=5";
> getContent(url)
> .data
> .unzip
> .runEncoded!((input) {
> foreach (record; input.data.parseJSON[1 .. $]) {
> // use or update record as desired
> }
> });
Eventually, something like this will be possible with jsoniopipe (I need
to update and release this too, it's probably broken with some of the
changes I just put into iopipe). Hopefully combined with some sort of
networking library you could process a JSON stream without reading the
whole thing into memory.
Right now, it works just like std.json.parseJSON: it parses an entire
JSON message into a DOM form.
-Steve
More information about the Digitalmars-d-learn
mailing list