uploading with curl

[Jonas Drewsen]

On Tuesday, 17 April 2012 at 07:07:06 UTC, Gleb wrote:
> Gentlemen,
>
> While working with curl library, I would like to discuss a few 
> more issues I faced with, if you don't mind.
>
> Here is the code sample I use:
>
>     auto f = new std.stream.BufferedFile("file.zip", 
> FileMode.In);
>     scope (exit) f.close();
>     auto client = FTP();
>     client.verbose(true);
>     client.url = "ftp://192.168.110.58/file.zip";
>     client.setAuthentication("user", "pass");
>     client.contentLength = cast(size_t)f.size;
>     client.handle.set(CurlOption.upload, 1L);
>     client.onSend = (void[] data)
>     {
>         return f.read(cast(ubyte[])data);
>     };
>     client.perform();
>
> 1. TIMEOUTS
>
> The code sample will work great if file.zip is small.
> If the file is big enough, we will get the following result:
>
>    * Connecting to 192.168.110.58 (192.168.110.58) port 1468
>    > TYPE I
>    < 200 Type set to I
>    > STOR file.zip
>    < 150 Opening BINARY mode data connection for file.zip
>    * Operation timed out after 120105 milliseconds with 
> 687751168 bytes received
>    * Closing connection #0
>    * Timeout was reached 
> std.net.curl.CurlTimeoutException at std\net\curl.d(3348): Timeout 
> was reached on handle 16BCB60
>
> So we were disconnected from server because of the timeout. To 
> avoid this behavior I use the following:
>
>    client.dataTimeout(dur!"weeks"(10));
>
> I'm not sure if we really need this kind of timeout so I use 
> the really big value. Anyway, I believe the default value of 
> 120 seconds is not enough for practical usage of file transfer 
> protocol. Maybe it's suitable for HTTP only?
>
> By the way, for some reason I can't use the value more then 10 
> weeks. In this case we can't even connect to server:
>
>    * About to connect() to 192.168.110.58 port 21 (#0)
>    *   Trying 192.168.110.58...
>    * connected
>    * Connected to 192.168.110.58 (192.168.110.58) port 21 (#0)
>    * server response timeout
>    * Closing connection #0
>    * Timeout was reached
>    std.net.curl.CurlTimeoutException at std\net\curl.d(3348): 
> Timeout was reached on handle 164CB60
>
> Of course, even bigger value is not necessary, but I'm not sure 
> this behavior is correct.

This is one of the many reasons I believe we should do our own
network library. Curl only support timeouts for connecting and
for the entire transfer. What you really want is better control
with sane defaults: DNS lookup timeout, connect timeout,
read/write activity timeout and timout for the entire operation.

Default "entire operation" timeout should be infinity and the
rest just some qualified guesses.

Anyway... I think the current default timeouts are how they
should be.

Regarding the 10 week limit you're mentioning please see the docs
for the Duration type:
http://dlang.org/phobos/core_time.html#Duration


>
> 2. PROGRESSBAR
>
> I've added the following to the above example:
>     client.onProgress = delegate int(size_t dlTotal, size_t 
> dlNow, size_t ulTotal, size_t ulNow)
>     {
>         return 0;
>     };
>
> When compiled and run we will see the following:
>
>     > STOR file.zip
>     < 150 Opening BINARY mode data connection for file.zip
>     * We are completely uploaded and fine
>     * Remembering we are in dir ""
>     < 226 Transfer complete
>     * Connection #0 to host 192.168.110.58 left intact
>     > QUIT
>     std.net.curl.CurlException at std\net\curl.d(3365): Progress 
> callback called on cleaned up Curl instance
>
> It looks like onProgress was called when the file was uploaded 
> and the data connection was closed. What am I doing wrong? 
> Should I return any other value when ulNow == ulTotal? In code 
> examples in the documentation onProgress does not return any 
> value, but it must.
>
> Maybe it would be better to use some simple progress bar by 
> default when curl.verbose was true? At least something like 
> this:
>
>     static const bStr = replicate("\b", 20);
>     static float perc;
>     client.onProgress = delegate int(size_t dlTotal, size_t 
> dlNow, size_t ulTotal, size_t ulNow)
>     {
>         perc = ulTotal ? cast(real)ulNow/cast(real)ulTotal*100 
> : 0;
>         writef("%s* %.2f%c uploaded", bStr, perc, '%');
>         return 0;
>     };
>
>
> Thank you in advance for your opinion on this issues.

You should always return 0 if you do not want to abort the job.
What you're describing sounds like a bug and I'll have a look at
it.

I think that including the progress in the verbose mode per
default will generate too much noise.

/Jonas