[OT] Efficient file structure for very large lookup tables?

[Francesco Cattoglio]

On Tuesday, 17 December 2013 at 19:09:49 UTC, H. S. Teoh wrote:
> Another OT thread to pick your brains. :)
>
> What's a good, efficient file structure for storing extremely 
> large
> lookup tables? (Extremely large as in > 10 million entries, 
> with keys
> and values roughly about 100 bytes each.) The structure must 
> support
> efficient adding and lookup of entries, as these two operations 
> will be
> very frequent.

What kind of key do you have? vector of floats? integers?
How about some uberfast compression algorithm (eg: LZ4) to 
compress pages when you have to offload data to the disk?