Encapsulating Locked File Appends

[Andrei Alexandrescu]

Sean Kelly wrote:
> dsimcha wrote:
>> == Quote from Sean Kelly (sean at invisibleduck.org)'s article
>>> dsimcha wrote:
>>>> Is there an easy way to write a function, or a function already 
>>>> written for
>>>> me, that will allow for a file shared between processes to be 
>>>> appended to
>>>> safely?  This should be done in a way that will make it impossible 
>>>> for two
>>>> processes to write to it at the same time, using locking, and should be
>>>> platform-independent.
>>>>
>>>> Performance does not matter because, in the use case I have, we're only
>>>> talking about one update every few minutes.  Simplicity, however, 
>>>> does matter.
>>>>  All I'm trying to do is run a simulation thousands of times on a 
>>>> bunch of
>>>> different computers sharing an NFS file system and have all of the 
>>>> results end
>>>> up in one nice plain text file instead of having each instance write 
>>>> to its
>>>> own file and having to keep track of them and piece them together by 
>>>> hand.
>>> Use a second file to represent the lock, with the contents as the lock
>>> owner.  While the file exists, poll.  When it's not there, create it and
>>> write the process id into it, then unlink the file when you're done.
>>
>> Wouldn't you have to somehow atomically poll and create the file?   
>> What if some
>> other process created the lock file between your call of exists() and 
>> write()?
> 
> If you use the "create only" flag when opening the file then it should 
> fail if the file was already created by someone else.  Unless NFS 
> doesn't provide a sufficiently reliable synchronization mechanism for 
> this to work, that is (I really don't know).

I've worked a lot with NFS and have the scars and learned the curses to 
prove it.

NFS is as non-deterministic as it gets when it comes about concurrent 
writes. There is next to no guarantee. The append problem is an absolute 
classic on NFS. I tried about five different schemes, all failed under 
mysterious circumstances. What I do now and suggest you do too is to 
have each different process create its own file. After all processes 
have ended, have a master process assemble all small files into one. 
It's really the only thing I got to work.


Andrei