std.process.execute performance without Config.inheritFDs

Jonathan Shamir via Digitalmars-d digitalmars-d at puremagic.com
Mon Jul 3 07:48:38 PDT 2017


Hey,

This code is from std.process:

         if (!(config & Config.inheritFDs))
         {
             import core.sys.posix.sys.resource;
             rlimit r;
             getrlimit(RLIMIT_NOFILE, &r);
             foreach (i; 3 .. cast(int) r.rlim_cur) close(i);
         }

This is a close-loop to make sure no fds are leaked before 
execve'ing, and can be disabled using Config.inheritFDs.

Here's a program I used to profile:

void main() {
     StopWatch sw;

     import core.sys.posix.sys.resource;
     rlimit r;
     errnoEnforce(getrlimit(RLIMIT_NOFILE, &r) == 0);
     writefln("The rlimit is %d (hard: %s)", r.rlim_cur, 
r.rlim_max);
     //r.rlim_cur = 1024 * 1024;
     //errnoEnforce(setrlimit(RLIMIT_NOFILE, &r) == 0);

     sw.start();
     foreach (_; 0 .. 100) {
         executeShell("echo hello", null, 
std.process.Config.inheritFDs);
     }
     sw.stop();

     writefln("Total time: %s", sw.peek().to!Duration.to!string);
}

These are the results I got:

// On a typical linux machine, without inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 3 secs, 875 ms, 759 μs, and 7 hnsecs

// with inheritFDs
The rlimit is 524288 (hard: 1048576)
Total time: 80 ms, 549 μs, and 2 hnsecs

// osx, without inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 476 ms, 483 μs, and 5 hnsecs

// with inheritFDs
The rlimit is 7168 (hard: 9223372036854775807)
Total time: 352 ms, 637 μs, and 7 hnsecs

So obviously this becomes a problem if the rlimit is high.

A few suggestions:
1. Make inheritFDs the default, since most people aren't aware of 
the performance cost. Also most programs will run just fine if a 
few fds are leaked.
1.1. Or, by default, close up to min(rlim_cur, 1024) - this 
should be enough for most processes. (If I'm not mistaken, that's 
what they do in python). Also this change won't break existing 
code.
2. All phobos fds should be opened with O_CLOEXEC, as intended. 
This eliminates the need to close all the fds when execve'ing.
3. Optimization for linux - use /proc to check which fds are 
actually opened, and close them, instead of making thousands of 
syscalls!


More information about the Digitalmars-d mailing list