Optimising Memory Usage

Query Habits

Select only the columns a query actually needs — unreferenced columns stay mapped on disk rather than being copied into RAM. The virtual column i (row index) forces materialisation when referenced, so use it only when an aggregation genuinely depends on row position.

Selecting fewer columns only avoids copying the columns you don't ask for — the ones you do ask for still get copied into a fresh in-memory result the moment a where constraint is involved, as covered in the Memory Mapping section. select price from t where date=x is cheap to read (only price is touched on disk), but its output is still a new heap-allocated table, sized by however many rows matched the filter — not something that stays mapped.

For multi-day operations like an as-of join across a date range, iterate date by date rather than selecting across every date at once — processing one date's worth of data at a time avoids building one huge temporary structure in memory, and tends to map more efficiently besides.

Joining two large lists or tables (,) needs room for the result alongside the original inputs while it runs, so peak memory during a big join can briefly run well above either input's own size on its own. For very large joins, this combined footprint — not the final result alone — is often what actually determines whether the operation fits in memory. Once the inputs are no longer needed, release them; .Q.gc[] can then return their memory to the operating system.

Architectural Habits

Splitting responsibilities across processes keeps any one process's memory footprint manageable: real-time capture, intraday analytics, and historical storage are commonly separated into different components rather than combined into one process holding everything. Some tables or columns can be pushed to disk directly instead of held in-memory for the whole day — useful for data that's rarely queried but expensive to keep resident.

Deleting rows from an in-memory table is expensive, since it has to rewrite every column vector — for workloads with frequent deletes, persisting to disk incrementally throughout the day (an "intraday" pattern) avoids that cost building up.

A slow subscriber is a common, easy-to-miss source of memory growth on a tickerplant: published messages queue up in that subscriber's outgoing buffer for as long as it lags behind, so one consumer falling behind under load can grow memory on the publisher side, not just its own. Watching consumer lag, not just the tickerplant's own memory, is part of keeping that growth in check.

Loading and Writing in Chunks

Loading or writing an entire file at once means holding all of it in memory simultaneously. .Q.fsn streams a file in chunks instead, and the same chunked approach applies to writers — CSV export, for example — to avoid a single in-memory blob holding the whole dataset.

A few smaller habits compound over time: avoid nesting data structures where a flatter encoding (e.g. a compact string format) would do, since nested structures cost more to store and process; and watch for sym file bloat in HDB processes, since every symbol ever seen gets loaded into memory on every process that touches that HDB — the cost multiplies across replicated/distributed deployments.

On well-tuned hardware (fast disks, NVMe arrays), a large HDB query's bottleneck can shift away from disk I/O entirely and onto memory-copy bandwidth instead — the speed at which data can be copied from the mapped disk buffer into a working buffer. That bandwidth depends on how many of a machine's RAM channels are actually populated; a system running with only one or two memory modules installed leaves most of its available copy bandwidth unused, regardless of how fast the underlying disks are.