randomIO, std.file, core.stdc.stdio

[Adam D. Ruppe via Digitalmars-d-learn]

On Tuesday, 26 July 2016 at 19:30:35 UTC, Charles Hixson wrote:
> It looks as if the entire file is stored in memory, which is 
> not at all what I want, but I also can't really believe that's 
> what's going on.


It is just mapped to virtual memory without actually being loaded 
into physical memory, so when you access the array it returns, 
the kernel loads a page of the file into memory, but it doesn't 
do that until it actually has to.

Think of it as being like this:

struct MagicFile {
     ubyte[] opIndex(size_t idx) {
           auto buffer = new ubyte[](some_block_length);
           fseek(fp, idx, SEEK_SET);
           fread(buffer.ptr, buffer.length, 1);
           return buffer;
     }
}


And something analogous for writing, but instead of being done 
with overloaded operators in D, it is done with the MMU hardware 
by the kernel (and the kernel also does smarter buffering than 
this little example).


> A part of the problem is that I don't want this to be a process 
> with an arbitrarily high memory use.

The kernel will automatically handle physical memory usage too, 
similarly to a page file. If you haven't read a portion of the 
file recently, it will discard that page, since it can always 
read it again off disk if needed, but if you do have memory to 
spare, it will keep the data in memory for faster access later.


So basically the operating system handles a lot of the details 
which makes it efficient.


Growing a memory mapped file is a bit tricky though, you need to 
unmap and remap. Since it is an OS concept, you can always look 
for C or C++ examples too, like herE: 
http://stackoverflow.com/questions/4460507/appending-to-a-memory-mapped-file/4461462#4461462