Another new io library

[Steven Schveighoffer via Digitalmars-d]

It's no secret that I've been looking to create an updated io library 
for phobos. In fact, I've been working on one on and off since 2011 (ouch).

After about 5 iterations of API and design, and testing out ideas, I 
think I have come up with something pretty interesting. It started out 
as a plan to replace std.stdio (and that did not go over well: 
https://forum.dlang.org/post/j3u0l4$1atr$1@digitalmars.com), in addition 
to trying to find a better way to deal with i/o. However, I've scaled 
back my plan of world domination to just try for the latter, and save 
tackling the replacement of Phobos's i/o guts for a later battle, if at 
all. It's much easier to reason about something new than to muddle the 
discussion with how it will break code. It's also much easier to build 
something that doesn't have to be a drop-in replacement of something so 
insanely complex.

I also have been inspired over the last few years by various great 
presentations and libraries, two being Dmitry's proof-of-concept library 
to have buffers that automatically move/fill when more data is needed, 
and Andrei's std.allocator library. They have changed drastically the 
way I have approached this challenge.

Therefore, I now have a new dub-based repository available for playing 
with: https://github.com/schveiguy/iopipe. First, the candy:

- This is a piping library. It allows one to hook buffered i/o through 
various processors/transformers much like unix pipes or range 
functions/algorithms. However, unlike unix pipes, this library attempts 
to make as few copies as possible of the data.

example:

foreach(line; (new IODevice(0)).bufferedInput
     .asText!(UTFType.UTF8)
     .byLine
     .asInputRange)
    // handle line

- It can handle 5 forms of UTF encoding - UTF8, UTF16, UTF16LE, UTF32, 
UTF32LE (phobos only partially handles UTF8). Sorry, no grapheme support 
or other utf-related things, but this of course can be added later.

- Arrays are first-class ioPipe types. This works:

foreach(line; "one\ntwo\nthree\nfour\n".byLine.asInputRange)

- Everything is compile-time for the most part, and uses lots of 
introspection. The intent is to give the compiler full gamut of 
optimization capabilities.

- I added rudimentary compression/decompression support using 
etc.c.zlib. Using compression is done like so:

foreach(line; (new IODevice(0)).bufferedInput
     .unzip
     .asText!(UTFType.UTF8)
     .byLine
     .asInputRange)

- The plan is for this to be a basis to make super-fast and modular 
parsing libraries. I plan to write a JSON one as a proof of concept. So 
all you have to do is add a parseJSON function to the end of any chain, 
as long as the the input is some pipe of text data (including a string 
literal).


=================

I will stress some very very important things:

1. This library is FAR from finished. Even the concepts probably need 
some tweaking. But I'm very happy with the current API/usage.

2. Docs are very thin. Unit tests are sparse (but do pass).

3. The focus of this library is NOT replacement of std.stream, or even 
low-level i/o in general. In fact, I have copied over my stream class 
from previous attempts at this i/o rewrite ONLY as a mechanism to have 
something that can read/write from file descriptors with the right API 
(located in iopipe/stream.d). I admit to never having looked at 
std.stream really, so I have no idea how it would compare.

4. As the stream framework is only for playing with the other useful 
parts of the library, I only wrote it for my OS (OSX), so you won't be 
able to play out of the box on Windows (probably can be added without 
much effort, or use another stream library such as this one that was 
recently announced: 
https://forum.dlang.org/post/xtxiuxcmewxnhseubyik@forum.dlang.org), but 
it will likely work on other Unixen.

5. This is NOT thread-aware out of the box.

6. There is a concept in here I called "valves". It's very weird, but it 
allows unifying input and output into one seamless chain. In fact, I 
can't think of how I could have done output in this regime without them. 
See the convert example application for details on how it is used.

7. I expect to be changing the buffer API, as I think perhaps I have the 
wrong abstraction for buffers. However, I did attempt to have a 
std.allocator version of the buffer.

8. It's not on code.dlang.org yet. I'll work on this.

Destroy!

-Steve