std.process: how to process stdout chunk by chunk without waiting for process termination

[Kevin Lamonte]

Timothee Cour wrote:

> On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer
> <schveiguy at yahoo.com>wrote:
> 
>> On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <
>> thelastmammoth at gmail.com> wrote:
>>
>>  I'd like to do the following:
>>>
>>> auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);
>>>
>>> while(true){
>>> version(A1)
>>>   string line=pipes.stdout.readln;
>>> version(A2)
>>>   auto line=pipes.stdout.readChunk(**10);
>>> version(A3)
>>>   auto line=pipes.stdout.readChar();
>>>
>>>   // do something with line
>>>
>>>   if(tryWait(pipes.pid).**terminated)
>>>     break;
>>> }
>>>
>>>
>>> The problem is that 'string line=pipes.stdout.readln;' seems to block
>>> until
>>> the process is terminated, ie if the command is a long running command
>>> that
>>> prints a line every 1 second for 10 seconds, this program will wait 10
>>> seconds before starting the processing.
>>> I also tried with rawRead, readf, fgetc but couldn't make it work.
>>> I'm on OSX, if that matters.
>>>
>>> Is there any way to achieve this?
>>>
>>
>> I think the issue is on the child process side.  If you are using
>> buffered I/O you have to flush the buffer.
>>
> 
> yes
> 
> 
>> For instance, if the child is using D writeln or C printf, and you are
>> using stdout, then it will only flush after writing 4096 bytes.  You can
>> flush early by calling flush on stdout, or fflush in C.  Note that C will
>> auto-detect if it is an interactive console, and flush via newlines
>> instead.  So running the same program from the console will flush every
>> line!
>>
>> Alternatively, you can set the flush policy to flush after every line.
>>  See here:
>>
>> https://developer.apple.com/**library/ios/#documentation/**
>> 
System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>
>>
>> And here:
>>
>> 
http://dlang.org/phobos/std_**stdio.html#.File.setvbuf<http://dlang.org/phobos/std_stdio.html#.File.setvbuf>
>>
>> -Steve
>>
> 
> Thanks, that does indeed work if I have source code for the child program
> and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after
> main. However I want to make it work without modifying source of child
> program.
> 
> I tried
> http://stackoverflow.com/questions/1401002/trick-an-application-into-
thinking-its-stdin-is-interactive-not-a-pipewith
> script:
> auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
> Redirect.stderr);
> that works but has issues : only buffers stdout, not stderr; and I may not
> want to redirect stderr to stdout; also it won't work in more complex
> cases, eg if program contains '|' etc, and it requires replacing \r\n by
> \n.
> 
> I tried replacing fork with forkpty inside std.process. That doesn't seem
> to work: calling isatty(1),isatty(2)   on child process still returns 0.
> Not sure why.
> 
> I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
> creation in std.process (after the fork with case 0); doesn't work either.

I ran into this also.  My solution is the makeShell() function at 
https://github.com/klamonte/d-tui/blob/master/tterminal.d .  I'd like to see 
forkpty in phobos myself, so I just opened bug 10464 for that request.