Why Anymap Speeds Up Compound Columns

Question

A table column holding dictionaries — one per row, each with a different shape — used to be slow to query from disk. What changed with anymap (kdb+ 3.6+), and why?

Solution

t:([] sym:`a`b; data:(`x`y!1 2;`z!3))
`:t/ set .Q.en[`:.;t]     / splay (enumerating symbols); data is stored as anymap
type get[`:t/]`data       / 77h

Explanation

Ordinary on-disk columns are memory-mapped because every value has the same fixed width — q can compute exactly where row n lives in the file without reading anything else. A column of dictionaries, nested lists, or other non-uniform structures doesn't have that property, so before anymap, the whole column had to be copied into RAM and deserialised just to read any of it.

Anymap solves this by giving each compound value a self-describing, mappable layout on disk, so q can page individual entries in on demand — the same lazy, on-access loading that ordinary fixed-width columns already get — instead of materialising the entire column up front. The practical effect can be large, since only the rows actually touched need to be read rather than the whole column.

One exception survives anymap: symbols inside an anymap structure are always copied onto the heap when accessed, never left mapped, because of how q interns symbols globally. Anymap only changes things for compound, non-uniform data.