Issues with std.net.curl on Win 10 x64

cptgrok sanden.shelton at gmail.com
Mon Mar 25 16:25:37 UTC 2019


I need to review syslogs for over 160 systems monthly, and I am 
trying to write a utility to automate bulk downloads from a 
custom web service where they are hosted. I need to calculate a 
date range for the prior month, add start and end date and a 
serial number to the query string for each system, which is easy, 
and in a foreach(system; systems) loop in main() I call a 
function passing the string url in to download and write a log to 
file. For a small number of systems, it works.

My trouble is, using std.net.curl, if I use get(URL) to get the 
entire text in a single call and write to file, memory usage 
spirals out of control immediately and within 20 or so calls, 
gets to about 1.3 GB and the program crashes. If I use 
byLineAsync(URL), then foreach(line; range) write the lines to 
file one at a time the memory usage never gets above 5MB but it 
just hangs always at the 51st call in the loop regardless of what 
parameters are in the query string, or how much data I have 
downloaded. The program never terminates, even after hours, but I 
can't see ANY activity on the process, CPU, mem or network. I can 
break my download jobs into <=50 systems (and it seems to work), 
but that seems like sweeping something under the rug, probably 
leading to future issues.

I'm using the 32bit binary from 
libcurl-7.64.0-WinSSL-zlib-x86-x64.zip on the release archive, 
and DMD 2.085.0. I've tried curl 7.63 and 7.57 but the behavior 
is the same.

Am I doing something wrong or is there some issue with curl or 
something else? I'm pretty new to D and I'm not sure if I need to 
go right down to raw sockets and re-invent the wheel or if there 
is some other library that can help. If I get this working, it 
could potentially save myself and many others hours per week.


More information about the Digitalmars-d-learn mailing list