Tips on TCP socket to postgresql middleware

[Ali Çehreli]

On 2/19/22 12:13, Chris Piker wrote:

 >    * gotchas you've run into in your multi-threaded (or just concurrent)
 > programs,

I use the exact scenario that you describe: Multiple threads process 
data and pass the results to a "writer" thread that persist it in a file.

The main gotcha is your thread disappearing without a trace. The most 
common reason is it throws an exception and dies.

Another one is to set the message box sizes to throttle. Otherwise, 
producers could produce more than the available memory before the 
consumer could consume it. Unlike the main thread, there is nobody to 
catch an report this "uncaught" exception.

   https://dlang.org/phobos/std_concurrency.html#.setMaxMailboxSize

You need to experiment with different number of threads, the buffer size 
that you mention, different lengths of message boxes, etc. For example, 
I could not gain more benefit in my program beyond 3 threads (but still 
set the number to 4 :p Humans are crazy.).

In case you haven't seen yet, the recipe for std.concurrency that works 
for me is summarized here:

   https://www.youtube.com/watch?v=dRORNQIB2wA&t=1735s

Ali