XML Benchmarks in D
Alexander Panek
alexander.panek at brainsware.org
Fri Mar 14 01:40:20 PDT 2008
BCS wrote:
> Reply to kris,
>
>> BCS Wrote:
>>
>>> what might be interesting is to make a version that works with slices
>>> of the file rather than ram. (make the current version into a
>>> template specialized on char[] and the new one on some new type?)
>>> That way only the parsed meta data needs to stay in ram. It would
>>> take a lot of games mapping stuff in and out of ram but it would be
>>> interesting to see if it could be done.
>>>
>> It would be interesting, but isn't that kinda what memory-mapped files
>> provides for? You can operate with files up to 4GB in size (on a 32bit
>> system), even with DOM, where the slices are virtual addresses within
>> paged file-blocks. Effectively, each paged segment of the file is a
>> lower-level slice?
>>
>
> Not as I understand it (I looked this up about a year ago so I'm a bit
> rusty). on 32bits, you can't map in 4GB because you need space for the
> programs code (and on windows you only get 3GB of address space as the
> OS gets that last GB) Also what about a 10GB file? My idea is to make
> some sort of lib that lest you handle larges data sets (64bit?) You
> would ask for a file to be "mapped in" and then you would get an object
> that syntactically looks like an array. Indexes ops would actually map
> in pieces, slices would generate new objects (with ref to the parent)
> that would, on demand, map stuff in. Some sort of GCish thing would
> start un mapping/moving stings when space gets tight. If you never have
> to actual convert the data to a "real" array you don't ever need to copy
> the stuff, you can just leave it in the file. I'm not sure it's even
> possible or how it would work, but it would be cool. (and highly useful)
I've got this strange feeling in my stomach that shouts out "WTF?!" when
I read about >3-4GB XML files. I know, it's about the "if" and "whens",
but still; if you find yourself needing such a beast of an XML file, you
might possibly think of other forms of data structuring (a database,
perhaps?).
More information about the Digitalmars-d
mailing list