Anymap

The Problem Anymap Solves

Ordinary on-disk columns are mappable because every value has the same fixed width — q can compute exactly where row n lives without reading anything else. A nested column — one where each row holds a list, string, dictionary, or table-shaped value rather than a single fixed-width value — doesn't have that property, so before kdb+ 3.6, the whole column had to be copied into RAM and deserialised just to read any of it.

What Anymap Changed

Anymap (kdb+ 3.6+) is the representation q uses for these mapped nested columns in a splayed table: type reports 77h for an anymap column, versus 0h for an ordinary list of lists held fully in RAM.

t:([] sym:`a`b; data:(`x`y!1 2;`z!3))
`:t/ set .Q.en[`:.;t]          / splay it (enumerating symbol columns first)
type get[`:t/]`data            / 77h — the nested column is mapped as anymap
key `:t                        / `s#`.d`data`data#`data##`sym

On disk it's written as up to three files: the main column file (holding offsets into the data), a # file holding the collapsed binary data itself, and a ## file for any symbols inside the structure. Individual entries can then be page-faulted in on demand, the same lazy loading ordinary fixed-width columns already get, instead of materialising the entire nested column up front.

Because only the entries a query touches need to be read, accessing a large nested column can be dramatically faster than loading and deserialising the whole column first.

Symbols are always copied, even with anymap. Whenever an anymap structure contains symbols, the ## file has to be de-enumerated back onto the heap on access — because of how q's global symbol interning works, symbols can't be left sitting mapped the way plain numeric or character data can. So anymap cuts heap use a lot for nested structures generally, but a nested column full of unique symbols won't get that benefit on the symbol portion specifically.

Trade-offs, and How It Compares to Alternatives

Anymap unlocks variable-schema, "document-style" columns and lets the same code work against in-memory and on-disk versions of a table. It comes with real costs, though: querying nested anymap data loses the built-in map-reduce optimisation regular select gets, adds complexity when distributing queries across multiple processes, and is generally a poor fit for very large, real-time capture systems where simplicity and throughput matter more than schema flexibility.

It's also not necessarily the fastest option for variable, per-record schemas — just the most flexible. When the set of fields is small and stable, modelling them directly as ordinary columns (a wider flat table), or as rows of a keyed or parted table, keeps all the usual column optimisations available. Anymap's advantage is flexibility and memory behaviour: because entries are mapped rather than deserialised up front, reading them doesn't require the whole column to be materialised on the heap — unlike, say, storing each row as serialised bytes (-8!) and deserialising every one on read (-9!).