Reading a structured binary file?

[Jonathan M Davis]

On Saturday, August 03, 2013 14:31:16 H. S. Teoh wrote:
> On Sat, Aug 03, 2013 at 02:25:23PM -0700, Jonathan M Davis wrote:
> > On Saturday, August 03, 2013 23:10:12 John Colvin wrote:
> > > On Saturday, 3 August 2013 at 18:23:58 UTC, Gary Willoughby wrote:
> > > > On Saturday, 3 August 2013 at 18:14:47 UTC, Gary Willoughby
> > > > 
> > > > wrote:
> > > >> This sounds a great idea but once the file has been opened as
> > > >> a MmFile how to i convert this to a ubyte[] so the
> > > >> std.bitmanip functions work with it?
> > > > 
> > > > I'm currently doing this:
> > > > 	auto file = new MmFile("file.dat");
> > > > 	ubyte[] buffer = cast(ubyte[])file[];
> > > > 	buffer.read!uint(); //etc.
> > > > 
> > > > Is this how you would recommend?
> > > 
> > > That defeats the object of memory mapping, as the [] at the end
> > > of cast(ubyte[])file[] implies copying the whole file in to
> > > memory.
> > 
> > Are you sure about that? Maybe I'm just not familiar enough with mmap,
> > but I don't see anything in MmFile which would result in it copying
> > the whole file into memory. I guess that I'll have to do some more
> > reading up on mmap.  Certainly, if slicing it like that copies it all
> > into memory, that's a big problem.
> 
> [...]
> 
> I think he meant that the OS will have to load the entire file into
> memory if you sliced the mmap'ed file, not that you'll copy all the
> data.
> 
> I'm not certain this is true, though, because slicing as I understand it
> only returns the address of the start of the mmap'ed addresses coupled
> with its length. I don't think the OS will actually load anything into
> memory until you reference an address within that mmap'ed range. And
> even then, only those disk blocks that correspond with the referenced
> addresses will actually be loaded -- this is the point of virtual
> memory, after all.

That's what I thought that mmap did, but it's not something that I've studied 
in detail.

Aside from that though, my main complaint about MmFile is the fact that it's a 
class when it really should be a reference-counted struct. At some point, we 
should probably create MMFile or somesuch which _is_ a reference counted 
struct and then deprecate MmFile. But if we do that, then we should be sure of 
whatever other changes the implementation needs and do those with it.

- Jonathan M Davis