What is the best way to use requests and iopipe on gzipped JSON file

[Steven Schveighoffer]

On 10/13/17 6:07 PM, Steven Schveighoffer wrote:
> On 10/13/17 4:30 PM, Andrew Edwards wrote:
>> On Friday, 13 October 2017 at 20:17:50 UTC, Steven Schveighoffer wrote:
>>> On 10/13/17 3:17 PM, Steven Schveighoffer wrote:
>>>
>>>> this should work (something like this really should be in iopipe):
>>>>
>>>> while(input.extend(0) != 0) {} // get data until EOF
>>>
>>> This should work today, actually. Didn't think about it before.
>>>
>>> input.ensureElems(size_t.max);
>>>
>>
>> No, it errored out:
>> std.json.JSONException at std/json.d(1400): Unexpected end of data. (Line 
>> 1:8192)
> 
> 
> I reproduced, and it comes down to some sort of bug when size_t.max is 
> passed to ensureElems.
> 
> I will find and eradicate it.
> 

I think I know, the buffered input source is attempting to allocate a 
size_t.max size buffer to hold the expected new data, and cannot do so 
(obviously). I need to figure out how to handle this properly. I 
shouldn't be prematurely extending the buffer to read all that data.

The while loop does work, I may change ensureElems(size_t.max) to do 
this. But I'm concerned about accidentally allocating huge buffers. For 
example ensureElems(1_000_000_000) works, but probably allocates a GB of 
space in order to "work"!

-Steve