[dmd-concurrency] Shutdown protocol

[Robert Jacques]

On Wed, 20 Jan 2010 12:20:09 -0500, Andrei Alexandrescu  
<andrei at erdani.com> wrote:

> I like this. All - any related thoughts?
>
> Truth be told we can fix the example by changing main:
>
> wait(spawn(&copy("in1", "out1")), spawn(&copy("in2", "out2")));
>
> which I think is entirely fair.
>
>
> Andrei

I like the updated example, but the updated shutdown protocol sounds like  
a race waiting to happen. For example, suppose you spawn a thread (A) and  
send it's tid to another thread (B) and then exit. The spawned thread dies  
and the other then runs into errors. So now B's racing on A's shutdown  
event and may have intended behavior one day and then throw a bunch of  
shutdown exceptions the next.
However, expanding on this concept, if tid was ref-counted, shutdown  
exceptions could be thrown correctly and at least the above race would be  
avoided.

>
> Michel Fortin wrote:
>> I know this has been discussed at length while I was sleeping, but I  
>> also see two problems with the shutdown protocol currently in the draft.
>>  First, it's a nice concept that the main thread waits for other  
>> threads to shutdown before exiting. It's a simple concept which allows  
>> other threads to finish their work before exiting. Except for this: any  
>> thread that requires communication to finish its work cannot exit  
>> properly. Another problem is that threads waiting a shutdown signal  
>> might not end until the very termination of the application.
>>  For instance, here is a modified version of Andrei's copy example  
>> which copies two files in parallel by putting each copy operation in a  
>> separate thread:
>>  	import std.algorithm, std.concurrency, std.stdio;
>>  	void main() {
>> 		spawn(&copy("in1", "out1"));
>> 		spawn(&copy("in2", "out2"));
>> 	}
>>  	void copy(string input, string output) {
>> 		File source = File(input, "r");
>>  		enum bufferSize = 1024 * 100;
>> 		auto tid = spawn(&writer, outputPath); // Read loop
>> 		foreach (immutable(ubyte)[] buffer; stdin.byChunk(bufferSize)) {
>> 			send(tid, buffer);
>> 		}
>> 	}
>> 	void writer(string output) {
>> 		File target = File(output, "w");
>>  		// Write loop
>> 		for (;;) {
>> 			auto buffer = receiveOnly!(immutable(ubyte)[])();
>> 			target.rawWrite(buffer);
>> 		}
>> 	}
>>  Given the current shutdown protocol, this wouldn't work at all. The  
>> shutdown state would make writer() exit before copy has a chance to  
>> read most of the input. True, you could just a 'join()' to make the  
>> main thread explicitly wait for the two copy threads it spawned, but  
>> this would leave a writer thread inactive in the background while the  
>> other copy finishes. Add more files to copy and you'll get a lot of  
>> waisted writer threads doing nothing until the program terminates.
>>  So I'd propose a simple change to the shutdown protocol to make it  
>> scale better: when any thread finishes, it puts all the threads it  
>> spawned in shutdown state and exits (except for the main thread witch  
>> must wait before exiting to keep the other threads running).
>>  This somehow creates a ownership hierarchy between threads:
>>  	main --> file copy thread 1 -> writer thread 1
>>              \-> file copy thread 2 -> writer thread 2
>>  What happens in the above example is this: main puts the two file copy  
>> threads in shutdown mode. When a file copy thread finishes reading, it  
>> puts its writer thread in shutdown mode, which then exits.
>>  I think this scales better.
>>  (Note: it should be possible to change the owner thread for cases  
>> where this hierarchy isn't appropriate.)
>>
> _______________________________________________
> dmd-concurrency mailing list
> dmd-concurrency at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-concurrency