stdio line-streaming revisited

[Dave]

Andrei Alexandrescu (See Website For Email) wrote:
> Bill Baxter wrote:
>> kris wrote:
>>> Andrei Alexandrescu (See Website For Email) wrote:
>>
>>> - Tango is for D programmers; not C programmers.
>>
>> D programmers sometimes like to call 3rd party code written in other 
>> languages, and pretty much any interop in D has to happen via C 
>> compatibility.  E.g. pyD.  So I'm guessing if my D code calls on some 
>> Python code that prints to the console that somewhere down the line 
>> that eventually ends up on C's stdout.  I could be wrong, but at least 

Well, it *all* ends up using file descriptor 0 on POSIX systems.

>> that's why I *think* Andrei and Walter keep saying that C 
>> compatibility is important.
> 
> Yes.

Please help me to understand this issue... C's stdio predefined set of file handles or streams are 
simply wrappers around "pipes" into and out of the console, which in itself is just a part of 
another separate program (the shell). For the _great majority_ of cases where a D program interacts 
with another program though a shell (which it will even if spawned via the POSIX exec() family for 
example), as long as Tango can read from and write -- and eventually flush if/as the underlying 
shell requires -- to the console, why would a D program really need to read and write to the exact 
same handles? The same *file descriptors as the shell, yes,* but file handles?

Basically what I'm saying is this: Let the shell handle this stuff as it sees fit; that's what it is 
there for!

For more complex high-throughput (http://cr.yp.to/qmail.html) or multi-threaded C programs that 
could use the stdio API, you almost have to use a closer to the metal API (like unistd 
read()/write()) anyway. Somewhere in the docs for Qmail, security and performance issues were the 
main reason why it was written with a complete replacement for stdio. IIRC, that developer really 
rails on C's stdio as outdated, and as something that drastically needs to be replaced. BTW, I 
myself have written programs that interact very well (and easily) with Qmail though C's stdio even 
though the only common bond is the POSIX file descriptors.

And for the minority interop case, as long as Tango I/O offers the flexibility to follow the same 
conventions as the client or server process regardless of shell conventions, I would think that 
would be good enough, as this code often needs to be "hand-crafted" -- at least in my experience --, 
or use the shell native API anyway (i.e.: read()/write()). I don't know much about Tango, but it 
actually looks like it offers much broader functionality than phobos in this regard (for example, 
the http classes).

Can I get a concrete and preferably common example of where the current Tango implementation fails 
for stdin/stdout/stderr in this regard?

Tango does not fail with the benchmark/example you provided... Similar functionality (at its base) 
as your program is by far the most common use case for reading and writing to the console.

> 
>> Andrei -- by "compatibility" does that mean if I rebind stdio/stdout 
>> to something different that both D and C's output go to the new 
>> place?  Or is it still necessary to rebind them individually?  I did 
>> this once for some legacy code in C++, and found that I had to rebind 
>> 3 things: the C streams, the C++ old-style streams from <iostream.h> 
>> (was under MSVC 6), and the new-style C++ streams from <iostream>.    
>> And then I had to do the interleaving myself, which didn't really work 
>> (because all the streams were just writing to output buffers 
>> individually).  If what you're talking about with compatibility would 
>> avoid that kind mess, that is certainly be a good thing.
> 
> Exactly. I'm hoping to save the D community from the iostreams mess, 
> which requires the programmer to deal with two (or three as in your 
> example) incompatible APIs. It's iostreams' amazing counterperformance 
> that they managed to cut the speed so effectively in the process :o).
> 
> I think a wise thing that Tango could do is to offer a 
> backward-compatible stdio, and a high-performance I/O class using the 
> latest and greatest from each I/O (zero-copy (see 
> http://research.sun.com/techrep/1995/abstract-39.html), asynchronous, 
> threaded, kernelized, teleported, you name it).
> 

I really don't think it needs to be that complicated, especially since Tango apparently performs 
very well anyhow for the most common cases.

> The main desideratum is that fast/incompatible I/O is opt-in, not lock-in.
> 

If phobos and Tango could be bundled together on equal footing with the D distributions, phobos 
could fill the 'stdio compatible' role.

- Where that is needed (the minority of cases, almost always developed by experienced programmers), 
D can easily do that. Tango and/or phobos could even provide a separate API for that, but not the 
default.
- The "C way" is not the "D way". D is supposed to be an improvement over C w/o all of the C++ 
corner cases. This paradigm should extend to the std. lib. as well.
- A lot of very experienced and smart people have contributed to Tango. I'd venture a guess that as 
a group they have all used C's stdio extensively and have also done various forms of interop in the 
past. They apparently don't have a problem with the current implementation, so I'd be very reluctant 
to question them on it w/o some very concrete examples of the most common use cases that fail with 
Tango.
- Speaking only for myself, at one time I was reluctant to accept anything like Tango simply because 
it seemed to be a split from Walter's direction, and because phobos was kind-of 'C like' so I was 
more comfortable with it. However, here is the Tango team willing and able to contribute, and more 
than happy (for the most part) to leave the compiler development and overall direction of the 
language to Walter. I think Walter's (and perhaps your?) time is best spent on that, and let the 
Tango team wrestle with the std. lib.

I'd like to suggest this: Distribute both phobos and Tango on equal footing with the D 
distributions, and have the compiler link both in by default (with a switch to not link only one):

- Walter and David Friedman could automate the build process to grab the latest Tango.
- Some overlapping areas (like the GC) would have to be resolved somehow (Tango team - how tough 
would this be?).
- Add the Tango docs. to the DigitalMars site on "equal footing" as well.
- C++ distributions provide somewhat of a template and precedence for distributing more than one 
library with the compiler.

Thanks,

- Dave

> 
> Andrei