learnkdb

Core Concepts

Memory Management & Memory Mapping

q's heap and allocation classes, memory-mapped files and the OS page cache, anymap, garbage collection, and practical tips for keeping a q process's memory footprint under control.

Overview

This topic looks under the hood: what the computer is doing while q runs a query, and where the memory goes. You don't need it to write q. But it explains the questions that otherwise feel mysterious:

  • Why is a query fast the second time I run it?
  • Why can q query a 2 TB database on a machine with 64 GB of RAM?
  • Why does top say q is using far more memory than my data?
  • Why didn't memory go down when I deleted a big table?
  • Why is selecting fewer columns so much faster?

The Big Picture

Everything in this topic happens somewhere in this picture:

CPU does the work, but can only work on data in its caches core 1L1 · L2 core 2L1 · L2 core 3L1 · L2 core 4L1 · L2 L3 cache, shared by all cores 64-byte cache lines · ~100 ns RAM q's heap objects q owns: in-memory tables, query results, … page cache (owned by the OS) copies of recently used file pages mapped into q HDB columns other files any program's other programs A q process uses its heap, plus the file pages it has mapped 4 KB pages · µs to ms Disk survives a restart HDB column files, tickerplant logs, everything saved
Data flows from disk, through RAM, into the CPU's caches. Each step up is faster and smaller than the one below.

Reading from the bottom up:

  • Disk holds everything permanently, such as HDB column files and tickerplant logs. It is by far the slowest place to read from.
  • RAM holds three kinds of data that matter here:
    • q's heap: objects q creates and owns, such as tables in memory, query results and temporary lists.
    • The page cache: copies of recently read file pages, kept by the operating system and shared by every program. When q memory-maps an HDB column, it reads those pages directly instead of copying them.
    • Everything else: other programs.
  • The CPU's caches hold the small amount of data the CPU is working on right now. Data moves into them from RAM in 64-byte pieces, which is why q's habit of storing each column as one long, contiguous vector is so fast.

Most "where did my memory go?" puzzles come from mixing up q's heap with the page cache. Most "why is this slow?" puzzles come from data having to travel further down the picture than expected.

How This Topic Is Organised

Read the sections in order. Each builds on the one before:

  1. Caches, Virtual Memory, and the Page Cache: the computer-science background, in plain English, with an experiment you can run in q.
  2. Memory Mapping (mmap): how q reads on-disk data without loading it, with an experiment that watches a process's memory change step by step.
  3. Memory Allocation (q's Heap): how q hands out memory for its own objects, and why sizes are rounded up.
  4. Garbage Collection & Reference Counting: when q frees memory, and when (and why not) it gives it back to the OS.
  5. Optimising Memory Usage and Anymap: practical habits, and how q maps columns of lists such as strings.

Keeping Memory Under Control

The habits that matter most in practice:

  • Read less. Filter on the partition column first, and select only the columns you need.
  • Watch what queries create, not just what they read. Results and temporary lists live in q's heap, even when the source data is mapped.
  • Work through large histories a date at a time rather than all at once.
  • Expect some operations to need room for both input and output, such as a large join or update, so their peak memory is higher than their result.
  • Know your numbers. .Q.w[] reports used (live q objects), heap (memory q has reserved), mmap (mapped files), and syms and symw (the symbol pool).

Sections