std.process: how to process stdout chunk by chunk without waiting for process termination

[Timothee Cour]

On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer
<schveiguy at yahoo.com>wrote:

> On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <
> thelastmammoth at gmail.com> wrote:
>
>  I'd like to do the following:
>>
>> auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);
>>
>> while(true){
>> version(A1)
>>   string line=pipes.stdout.readln;
>> version(A2)
>>   auto line=pipes.stdout.readChunk(**10);
>> version(A3)
>>   auto line=pipes.stdout.readChar();
>>
>>   // do something with line
>>
>>   if(tryWait(pipes.pid).**terminated)
>>     break;
>> }
>>
>>
>> The problem is that 'string line=pipes.stdout.readln;' seems to block
>> until
>> the process is terminated, ie if the command is a long running command
>> that
>> prints a line every 1 second for 10 seconds, this program will wait 10
>> seconds before starting the processing.
>> I also tried with rawRead, readf, fgetc but couldn't make it work.
>> I'm on OSX, if that matters.
>>
>> Is there any way to achieve this?
>>
>
> I think the issue is on the child process side.  If you are using buffered
> I/O you have to flush the buffer.
>

yes


> For instance, if the child is using D writeln or C printf, and you are
> using stdout, then it will only flush after writing 4096 bytes.  You can
> flush early by calling flush on stdout, or fflush in C.  Note that C will
> auto-detect if it is an interactive console, and flush via newlines
> instead.  So running the same program from the console will flush every
> line!
>
> Alternatively, you can set the flush policy to flush after every line.
>  See here:
>
> https://developer.apple.com/**library/ios/#documentation/**
> System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>
>
> And here:
>
> http://dlang.org/phobos/std_**stdio.html#.File.setvbuf<http://dlang.org/phobos/std_stdio.html#.File.setvbuf>
>
> -Steve
>

Thanks, that does indeed work if I have source code for the child program
and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main.
However I want to make it work without modifying source of child program.

I tried
http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipewith
script:
auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
Redirect.stderr);
that works but has issues : only buffers stdout, not stderr; and I may not
want to redirect stderr to stdout; also it won't work in more complex
cases, eg if program contains '|' etc, and it requires replacing \r\n by \n.

I tried replacing fork with forkpty inside std.process. That doesn't seem
to work: calling isatty(1),isatty(2)   on child process still returns 0.
Not sure why.

I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
creation in std.process (after the fork with case 0); doesn't work either.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d-learn/attachments/20130619/44174158/attachment.html>