What is the best way to use requests and iopipe on gzipped JSON file
Steven Schveighoffer
schveiguy at yahoo.com
Fri Oct 13 22:18:57 UTC 2017
On 10/13/17 6:07 PM, Steven Schveighoffer wrote:
> On 10/13/17 4:30 PM, Andrew Edwards wrote:
>> On Friday, 13 October 2017 at 20:17:50 UTC, Steven Schveighoffer wrote:
>>> On 10/13/17 3:17 PM, Steven Schveighoffer wrote:
>>>
>>>> this should work (something like this really should be in iopipe):
>>>>
>>>> while(input.extend(0) != 0) {} // get data until EOF
>>>
>>> This should work today, actually. Didn't think about it before.
>>>
>>> input.ensureElems(size_t.max);
>>>
>>
>> No, it errored out:
>> std.json.JSONException at std/json.d(1400): Unexpected end of data. (Line
>> 1:8192)
>
>
> I reproduced, and it comes down to some sort of bug when size_t.max is
> passed to ensureElems.
>
> I will find and eradicate it.
>
I think I know, the buffered input source is attempting to allocate a
size_t.max size buffer to hold the expected new data, and cannot do so
(obviously). I need to figure out how to handle this properly. I
shouldn't be prematurely extending the buffer to read all that data.
The while loop does work, I may change ensureElems(size_t.max) to do
this. But I'm concerned about accidentally allocating huge buffers. For
example ensureElems(1_000_000_000) works, but probably allocates a GB of
space in order to "work"!
-Steve
More information about the Digitalmars-d-learn
mailing list