Issues with std.net.curl on Win 10 x64

Boris Carvajal boris2.9 at gmail.com
Mon Mar 25 23:02:18 UTC 2019


On Monday, 25 March 2019 at 16:25:37 UTC, cptgrok wrote:
> Am I doing something wrong or is there some issue with curl or 
> something else? I'm pretty new to D and I'm not sure if I need 
> to go right down to raw sockets and re-invent the wheel or if 
> there is some other library that can help. If I get this 
> working, it could potentially save myself and many others hours 
> per week.

There is a limit of 50 concurrent messages per thread [1] in 
byLineAsync also the transmitBuffers argument takes part in.
So using multiple byLineAsync at same time/thread is going to 
block the process, I'm not sure if this is a bug or is by design.

You could use download() in a parallel foreach, something like 
this:

import std.stdio;
import std.parallelism;
import std.net.curl;
import std.typecons;

void main()
{
     auto connections = 3; // 3 parallel downloads
     defaultPoolThreads(connections - 1);
     auto retries = 4; // try up to 4 times if it fails
     auto logList = [
                 tuple("dlang.org", "log1.txt"), 
tuple("dlang.org", "log2.txt"),
                 tuple("dlang.org", "log3.txt"), 
tuple("dlang.org", "log4.txt"),
                 tuple("dlang.org", "log5.txt"), 
tuple("dlang.org", "log6.txt")];

     foreach (log; parallel(logList, 1))
     {
         HTTP conn = HTTP();

         foreach (i; 0 .. retries)
         {
             try
             {
                 writeln("Downloading ", log[0]);
                 download(log[0], log[1], conn);

                 if(conn.statusLine.code == 200)
                 {
                     writeln("File ", log[1], " created.");
                     break;
                 }
             }
             catch (CurlException e)
             {
                 writeln("Retrying ", log[0]);
             }
         }
     }
}

[1] 
https://github.com/dlang/phobos/blob/master/std/net/curl.d#L1679


More information about the Digitalmars-d-learn mailing list