Multithreaded file IO?

[Jonathan M Davis]

On Friday, September 23, 2011 23:01:17 Jerry Quinn wrote:
> Hi folks,
> 
> I wasn't sure whether this should go here or in the D devel list...
> 
> I'm trying to port a program where threads read from a file, process the
> data, then write the output data.  The program is cpu-bound.  In C++ I can
> do something like this:
> 
> class QueueIn {
>   ifstream in;
>   mutex m;
> 
>   string get() {
>     string s;
>     m.lock();
>     getline(in, s);
>     m.unlock();
>    return s;
>   }
> };
> 
> class QueueOut {
>   ofstream out;
>   mutex m;
>   void put(string s) {
>     m.lock();
>     out.write(s);
>     m.unlock();
>   }
> };
> 
> 
> In D, I'm so far having trouble figuring out the right idiom to do what I
> want.  I looked at std.parallel, but it seems a bit tricky to make my stuff
> work in this setting.  A std.stdio File cannot be part of shared class.
> 
> How would you do this with the latest D2?

A direct rewrite would involve using shared and synchronized (either on the 
class or a synchronized block around the code that you want to lock). However, 
the more idiomatic way to do it would be to use std.concurrency and have the 
threads pass messages to each other using send and receive.

So, what you'd probably do is spawn 3 threads from the main thread. One would 
read the file and send the data to another thread. That second thread would 
process the data, then it would send it to the third thread, which would write 
it to disk.

Unfortunately, I'm not aware of any good code examples of this sort of thing 
online. TDPL has some good examples, but obviously you'd have to have the book 
to read it. Given some time, I could probably cook up an example, but I don't 
have anything on hand.

- Jonathan M Davis