Phobos Wish List/Next in Review Queue?

[Johannes Pfau]

dsimcha wrote:
>Now that we've got a lot of contributors to Phobos and many projects
>in the works, I decided to start a thread to help us make a rough plan
>for Phobos's short-to-medium term development.  There are three goals
>here:
>
>1.  Determine what's next in the review queue after std.csv (voting on 
>std.csv ends tonight, so **please vote**).
>
>2.  Come up with a wish list of high-priority modules that Phobos is 
>missing that would make D a substantially more attractive language
>than it is now.
>
>3.  Figure out who's already working on what from the wish list and
>what bottlenecks, if any, are getting in the way and what can be done
>about them.
>
>The following is the wish list as I see it.  Please suggest additions 
>and correct any errors, as this is mostly off the top of my head.
>Also, status updates if you're working on any of these and anything 
>substantial has changed would be appreciated.
>
>*  Some higher level networking support, such as HTTP, FTP, etc.
>(Jonas Drewsen's CURL wrapper handles a lot of this and may be ready
>for a second round of review.)
>
>*  Serialization.  (Jacob Carolberg's Orange library might be a good 
>candidate.  IIRC he said it's close to ready for review.)
>
>*  Encryption and hashing.  (This is more an implementation problem
>than a design problem and AFAIK noone is working on it.)
>
>*  Containers.  (AFAIK noone is working on this.  It's tough to get 
>started because, despite lots of discussion at various times on this 
>forum, noone seems to really know what they want.  Since the
>containers in question are well-known, it's much more a design problem
>than an implementation problem.)
>
>*  Allocators.  (I think Phobos desperately needs a segmented 
>stack/region based allocator and I've written one.  I've also tried to 
>define a generic allocator API, mostly following Andrei's suggestions, 
>but I'll admit that I didn't really know what I was doing for the 
>general API.  Andrei has suggested that allocators should have 
>real-world testing on containers before being included in Phobos. 
>Therefore, containers block allocators and if the same person doesn't 
>write both, there will be a lot of communication overhead to make sure 
>the designs are in sync.)
>
>*  Streams.  (Another item where the bottleneck is mostly at the
>design level and people not really knowing what they want.)
>
>*  Compression/archiving.  (Opening standard compressed/archived file 
>formats needs to just work.  This includes at least zip, gzip, tar and 
>bzip2.  Of course, zip already is available and gzip is supported by
>the zlib module but with a crufty C API.  At least gzip and bzip2,
>which are stream-based as opposed to file-based, should be handled via
>streams, which means that streams block compression/archiving.  Also,
>since tar and zip are both file based, they should probably be handled
>by the same API, which might mean deprecating std.zip and rewriting
>it.)

AFAIK almost all compression (and encryption) algorithms work like
streams. The directory structure is often simply added on top of the
streaming functionality (at least true for all .tar.* formats).

BTW: The unfinished stream API blocks lots of projects: Encryption,
Compression, the curl wrapper could be integrated with streams, http
servers and almost everything else related to I/O.

>
>*  An improved std.xml.  (I think Thomas Sowinski is working on a 
>replacement, but I haven't seen any updates in a long time.)
>
>*  Matrices and linear algebra.  (Cristi Cobzarenco's GSoC project is
>a good starting point but it needs polish.  I've been in contact with
>him occasionally since GSoC ended and he indicated that he wants to
>get back to working on it but doesn't have time.  I've contributed to
>it sparingly, but find it difficult because I haven't gotten around to 
>familiarizing myself with the implementation details yet, and it's
>hard to get into a project that complex with a few hours a week as
>opposed to focusing full time on it.)
>
>*  std.database.  (Apparently Steve Teale is working on this.  This is
>a large, complicated project because we're trying to define a common
>API for a variety of RDBMSs.  Again, it's more a design problem than
>an implementation problem.)
>
>*  Better support for creating processes/new std.process.  (Lars 
>Kyllingstad wrote a replacement candidate for Posix and Steve 
>Schveighoffer ported it to Windows, but issues with the DMC runtime 
>prevent it from working on Windows.)
>
>*  Parallel algorithms.  (I've implemented a decent amount of these in 
>my std.parallel_algorithm Github project, but I've become somewhat 
>frustrated and unmotivated to finish this project because so many of
>the relevant algorithms seem memory bandwidth bound and aren't
>substantially faster when parallelized than when run serially.)

Inter-process communication, but as we have thrift now, that might be
ok. (Extending std.concurrency message passing to interprocess
communication with a low overhead serializer like messagepack would
still be great though)

Logging: AFAIK the new log module should be almost ready. I already
used it a few times, but you have to a patch druntime to use
it.

High performance IO: A reactor / proactor framework

A replacement for std.json

>
>After writing this, the general pattern I notice is that lots of stuff 
>is blocked by design, not implementation.  In a lot of cases people 
>don't really know what they want and analysis paralysis results.


-- 
Johannes Pfau