[OT] Efficient file structure for very large lookup tables?

[H. S. Teoh]

On Wed, Dec 18, 2013 at 01:14:58AM +0100, Francesco Cattoglio wrote:
> On Tuesday, 17 December 2013 at 19:09:49 UTC, H. S. Teoh wrote:
> >Another OT thread to pick your brains. :)
> >
> >What's a good, efficient file structure for storing extremely large
> >lookup tables? (Extremely large as in > 10 million entries, with keys
> >and values roughly about 100 bytes each.) The structure must support
> >efficient adding and lookup of entries, as these two operations will
> >be very frequent.
> 
> What kind of key do you have? vector of floats? integers?

Vector of (relatively small) integers.


> How about some uberfast compression algorithm (eg: LZ4) to compress
> pages when you have to offload data to the disk?

That's a good idea, actually, since the keys are very similar to each
other, so should compress very well when many are placed together in a
single page.


T

-- 
The diminished 7th chord is the most flexible and fear-instilling chord. Use it often, use it unsparingly, to subdue your listeners into submission!