Reading a structured binary file?

[Jonathan M Davis]

On Monday, August 05, 2013 23:04:58 H. S. Teoh wrote:
> On Tue, Aug 06, 2013 at 06:48:12AM +0200, Jesse Phillips wrote:
> [...]
> 
> > The only way I'm seeing to advance through the file is to keep an
> > index on where you're currently reading from. This actually works
> > perfect for the FileRange I mentioned in the previous post. Though
> > I'm not familiar with how mmfile manages its memory, but hopefully
> > there isn't buffer reuse or storing the slice could be overridden
> > (not an issue for value data, but string data).
> 
> I don't know about D's Mmfile, but AFAIK, it maps directly to the OS
> mmap(), which basically maps a portion of your program's address space
> to the data on the disk. Meaning that the memory is managed by the OS,
> and addresses will not change from under you.
> 
> In the underlying physical memory, pages may get swapped out and reused,
> but this is invisible to your program, since referencing them will cause
> the OS to swap the pages back in, so you'll never end up with invalid
> pointers. The worst that could happen is the I/O performance hit
> associated with swapping. Such is the utility of virtual memory.

mmap is awesome. It makes handling large files _way_ easier, especially when 
you have to worry about performance. It was a huge performance boost for one 
of our video recorder programs where I work when we switched to using mmap on 
it (this device is recording multiple video streams from cameras 24/7, and 
performance is critical). Trying to do what mmap does on your own is 
incredibly bug-prone and bound to be worse for performance (since you're doing 
it instead of the kernel). One of our older products tries to do it on its own 
(probably because the developers didn't know about mmap), and it's a royal 
mess.

- Jonathan M Davis