[Issue 24524] New: Very slow process fork if RLIMIT_NOFILE is too high

[d-bugmail at puremagic.com]

https://issues.dlang.org/show_bug.cgi?id=24524

          Issue ID: 24524
           Summary: Very slow process fork if RLIMIT_NOFILE is too high
           Product: D
           Version: D2
          Hardware: x86_64
                OS: Linux
            Status: NEW
          Severity: enhancement
          Priority: P1
         Component: phobos
          Assignee: nobody at puremagic.com
          Reporter: trikkuz at gmail.com

In `std.process`, the `spawnProcessPosix` function, used by all functions that
start a process, has a flag `Config.Flags.inheritFDs`. If set to false, it
reads the maximum limit of files that can be opened by a process,
`RFILE_NOFILE`, and then creates a structure for each one and finally iterates
through them.

Normally, the `RFILE_NOFILE` limit is set to 1024 or anyway a relatively low
number. The hard limit depends on the system; in some, it is 2^20, in others
even 2^30.

When starting a container with Docker, the limit is raised to the maximum
allowed, and this inevitably slows down the process visibly (because it has to
allocate 2^20 (or 2^30) elements and then iterate through them. An user
reported that on a system with 2^30 as the hard limit, it becomes literally
unusable.

There would be a faster way to do this by reading the open file descriptors
from `/dev/fd` with the code below, but we are inside a `@nogc` block, so
another solution must be found.

`
import std.file : exists, isDir;

if (exists("/dev/fd") && isDir("/dev/fd"))
{
    import std.file : dirEntries, SpanMode;
    import std.path : baseName;
    import std.algorithm : map, filter;
    import std.array : array;
    import std.string : indexOfNeither;
    import std.conv : to;

    // List all file descriptors except stdin, stdout, stderr, and the pipe
    auto fds = dirEntries("/dev/fd", SpanMode.shallow)
        .map!(d => d.name.baseName)
        .filter!(d => d.indexOfNeither("0123456789") < 0)
        .map!(d => d.to!int)
        .filter!(d => d >=3 && d != forkPipeOut)
        .array; // Copy to avoid closing while iterating

    // Close only the file descriptors that are open.
    foreach(fd; fds)
        close(fd);
}
`

--