Persisting Tables

Overview

kdb+ supports four storage formats, each optimised for a different data volume and query pattern. Choosing the right format determines whether your historical queries take milliseconds or minutes — and whether your process uses 1 GB or 100 GB of RAM to serve them.

The formats form a progression from simplest to most powerful:

• Flat — a single binary file; the whole table loads into memory on access. For small reference data.
• Splayed — one file per column inside a directory; columns load on demand. For large single-period tables.
• Partitioned — splayed tables inside date/int subdirectories; queries prune partitions automatically. For any time series database.
• Segmented — partitions distributed across multiple disks; parallel reads via secondary threads. For very high-volume data.

How This Topic Is Organised

The full breakdown — when to use each storage type, the 100k-row rule of thumb, partition column restrictions, adding data to an existing partition, and when segmentation becomes necessary — is in Storage Types under the HDB topic, which is where these decisions live in a production system.

The sections below cover the mechanics of the two foundational formats:

• How to write, read, and append to a splayed table: Appending to Splayed Tables.

Key Things to Know

The trailing / on a path is what tells kdb+ to write in splayed form:

`:trade/   set .Q.en[`:.] trade     / splayed — one file per column
`:trade.bin set trade                / flat — single binary file

.Q.en is required when splaying any table that contains symbol columns. It enumerates the symbols against the database's sym file, so each symbol column is stored as integer indices into that shared list. Without it, kdb+ refuses to splay the table: set fails with 'type.

After writing, columns are memory-mapped on access — the operating system loads only the pages needed to satisfy the query. A query that needs only a small part of a 10 GB column file reads far less than 10 GB from disk.