How to get output of piped process?

[Jedi]

On Wednesday, 17 February 2021 at 14:36:58 UTC, Steven 
Schveighoffer wrote:
> On 2/17/21 1:58 AM, Jedi wrote:
>> I an using pipeShell, I have redirected stdout, stderr, and 
>> stdin.
>> 
>> I am trying to read from the output and display it in my app. 
>> I have followed this code almost exactly except I use try wait 
>> and flush because the app is continuously updating the output. 
>> (it outputs a progress text on the same line and I'm trying to 
>> poll it to report to the user)
>> 
>> 
>> auto pipes = pipeProcess("my_application", Redirect.stdout | 
>> Redirect.stderr);
>> scope(exit) wait(pipes.pid);
>> 
>> // Store lines of output.
>> string[] output;
>> foreach (line; pipes.stdout.byLine) output ~= line.idup;
>> 
>> // Store lines of errors.
>> string[] errors;
>> foreach (line; pipes.stderr.byLine) errors ~= line.idup;
>> 
>> 
>> My code
>> 
>> auto p = pipeShell(`app.exe "`~f.name~`"`, Redirect.stdout | 
>> Redirect.stdin | Redirect.stderr);
>> 
>> 
>>              while(!tryWait(p.pid).terminated)
>>              {
>>                  string[] output;
>>                  foreach (line; p.stdout.byLine)
>
> You need to be careful here -- this will wait until stdout is 
> *closed*.
>
>>                  {
>>                      output ~= line.idup;
>>                      writeln(line);
>>                  }
>> 
>>                  string[] errors;
>>                  foreach (line; p.stderr.byLine)
>>                  {
>>                      errors ~= line.idup;
>>                      writeln("Err:"~line);
>>                  }
>
> Same thing here.
>
>>              }
>> 
>> wait(p.pid);
>> 
>> None of this works though. What is strange is that when I 
>> close out the debugger the app starts working(no console 
>> output but I able to see that it is doing something) but is 
>> very slow.
>> 
>> auto p = executeShell(`app.exe "`~f.name~`"`);
>> 
>> Does work, except I have no output or input. I have another 
>> app that I do the exact same code and I can get the output and 
>> parse it, but this is after the app terminates. I imagine the 
>> issue here is that I'm trying to get the output while the app 
>> is running.
>
> Without knowing the pattern of what your app is outputting, 
> it's hard to tell what will happen.
>
> The most common problem with people dealing with piped output 
> is not reading data off the pipe, which then makes the child 
> process hang trying to write to the pipe, because the buffer is 
> full.
>
> For instance, if your process outputs tons of stuff to stderr, 
> you will hang, because you are waiting for stdout to be closed 
> first before you read anything from stderr, the child process 
> fills up stderr pipe, and is put to sleep waiting for it to be 
> writable, never closing stdout.
>
> Unfortunately, std.process wraps all the pipes in File structs, 
> so you have almost no good mechanisms to properly read the data.
>
WTF?

> -Steve

Seriously, I can't simply get the output in real time? Come on, 
that is lame, Surely D can do better than that? How hard is it to 
get a buffer? Is there any hacks? How can one communicate with an 
app using std io if one can't actually communicate until the app 
is closed? It makes no sense.

But note that even executeShell doesn't display the output of the 
app.exe so it is more than just pipeShell.

The app just outputs text, just like almost every other text. One 
shouldn't have to know any pattern, that defeats the purpose. I 
should just be able to get the output of the app.exe, and also if 
the app is requesting input. This isn't rocket science but it 
seems someone wants to turn it in to it?

When the app.exe is running it just prints stuff out, every once 
in a while it might ask for input(e.g., to overwrite the file if 
it exists, but I can get around that by checking in D)... but 
ultimately I just want to consolidate the output it gives so I 
need access to it BEFORE the app closes. The app.exe processes 
files, takes some time to do so so if I have to wait to display 
anything nothing will be displayed for a long time.