Writing Partitioned Tables to Disk

The Write Path for a Date-Partitioned HDB

Writing in-memory data to a date-partitioned HDB involves three concerns that must be handled together: the directory structure, symbol enumeration, and the column sort order and attributes. Getting any one wrong produces a database that either fails to load or queries slowly.

Path Construction

Each date partition is a subdirectory of the HDB root containing a further subdirectory per table. kdb+ requires a trailing / in the path to signal splayed (column-per-file) storage:

hdbRoot: `:/data/hdb

/ Manual path construction:
tblPath: ` sv hdbRoot,`$string .z.d,`t,`   / `:/data/hdb/2025.01.15/t/

/ Using .Q.par, which returns the path WITHOUT the trailing slash...
.Q.par[hdbRoot; .z.d; `t]                  / `:/data/hdb/2025.01.15/t
/ ...so add one before using it with set
tblPath: ` sv .Q.par[hdbRoot; .z.d; `t],`

.Q.par also handles segmented databases (it picks the right segment from par.txt), which is why it is the usual way to build a partition path. For functions that operate on an existing splayed table — xasc, amend with @, key — the path without the slash works as it is.

Symbol Enumeration with .Q.en

Any table containing symbol columns must be passed through .Q.en before writing. This enumerates the symbols against the database's master sym file, replacing arbitrary symbol atoms with integer indices:

tblPath set .Q.en[hdbRoot; data]

Without .Q.en, each partition's sym column would be a disconnected list of symbols rather than indices into a shared enumeration. Cross-partition queries and where sym=`AAPL filters would fail or return wrong results.

Sort Order and the p# Attribute

Partitioned HDB queries benefit enormously from a parted (p#) attribute on the symbol column. The attribute tells kdb+ that all rows for a given symbol are contiguous — it can skip non-matching groups without scanning every row.

The attribute is only valid when the data is actually sorted by sym (and time within each sym group):

/ Apply parted attribute after sorting
@[`sym`time xasc tblPath; `sym; `p#]

This must be done either in memory before writing, or on the on-disk table immediately after writing.

Compression

kdb+ supports per-column compression using a 4-element write descriptor (path; logicalBlockSize; algorithm; zipLevel):

/ Write with gzip compression (algorithm=2, level=6)
(colPath; 17; 2; 6) set get colPath

Compression can be applied column-by-column after the initial write, or globally for all subsequent writes in a process by setting .z.zd:

.z.zd: 17 2 6   / all writes in this session will use these settings

Use -21! colPath at any time to inspect the compression metadata for an existing column file.