Memory Mapping (mmap)

Memory mapping is how q reads data stored on disk, such as the columns of an HDB, without loading whole files into memory. It builds directly on the previous section: virtual memory, pages, page faults and the page cache.

Two Ways to Read a File

A program can get at a file's data in two ways:

• Reading the file copies its bytes, all of them, into the program's own memory. The data is now in RAM twice: once in the page cache and once in the program.
• Mapping the file asks the OS to give the file a range of the program's addresses. Nothing is copied. When the program reads one of those addresses, a page fault brings in just that page, and the program reads it directly from the page cache.

When q opens an on-disk column, for example in an HDB or a splayed table, it maps it.

Mapping Is Not Loading

Mapping a file only creates entries in the page table. It's like being handed a map of a library: you know where every book is, but you haven't fetched any.

The data arrives one page at a time, when a query actually reads it. A query that touches 50 MB of a 2 TB file brings in about 50 MB, and the rest stays on disk.

Watch It Happen

This experiment maps a 763 MB file and watches the process's real memory use (RSS) at each step. It runs on Linux, because it reads RSS from /proc. It writes an 800 MB file into a demo folder (and, later on this page, a small tab folder), so delete them afterwards.

/ RSS of this q process, in MB, read from Linux's /proc
rss:{("J"$first system "awk '/VmRSS/{print $2}' /proc/",string[.z.i],"/status") div 1024}
mb:{x div 1048576}

`:demo/price set 100000000?100.0     / 100 million floats: 763 MB on disk
.Q.gc[];

p:get `:demo/price                    / map the file
(rss[]; mb .Q.w[]`used; mb .Q.w[]`mmap)

\t sum p                              / read every value, and time it
(rss[]; mb .Q.w[]`used; mb .Q.w[]`mmap)
\t sum p                              / and time it again

These are the results we got (yours will be similar):

What the numbers show:

• get read nothing. RSS stayed at 6 MB, even though q reports 762 MB mapped.
• sum p touched every page, so every page became part of q's resident memory. RSS rose by the size of the file.
• q's heap didn't grow at all. Those 762 MB are the file's pages in the page cache, mapped into q. They aren't q objects, and the OS can drop them again if it needs the RAM, because it can always re-read them from the file.

The two timings differ too. We got 82 ms for the first sum p and 52 ms for the second. The first run paid for a page fault on every page. Here the file had only just been written, so its pages were already in the page cache and these were all minor faults. On a cold file, read from disk, the first run would be much slower.

Results Are Copied, Not Mapped

The data a query reads can stay mapped, but anything a query produces is a new object in q's heap:

big:p where p>99.0     / about 1% of the values
count big
mb .Q.w[]`used

The result has about 1 million floats (8 MB), and used grows by about 8 MB. While the query ran, though, RSS rose by about 115 MB. p>99.0 builds a temporary boolean list with one byte for each of the 100 million values, about 100 MB, and q keeps that memory in its heap for reuse after the query finishes (see Garbage Collection).

So with mapped data, think about two costs:

• What a query touches, which costs page faults and page-cache space.
• What a query creates, which costs q heap: its result, and any temporary lists it builds on the way.

HDB Queries Only Touch What They Need

An HDB stores each column of each day in its own file. A query maps and reads only the files it needs:

This is why the most effective way to speed up an HDB query is usually to read less:

• Filter on the partition column first (where date=... or date within ...), so whole directories are skipped.
• Select only the columns you need. select price from trade where ... reads two files per day. select from trade where ... reads every column.

With the parted attribute on sym, which is standard in an HDB, where sym=`AAPL goes further: q reads only the part of each column that holds AAPL's rows (see Attributes).

Mapped Data Is Read-Only

q treats mapped files as read-only. Changing a mapped column, for example with update in memory, creates a new in-memory copy in q's heap, and the file on disk is unchanged. To change data on disk, you write it back explicitly, for example with set, or with @ on a file path.

Immediate and Deferred Mapping

For a splayed table (a directory with one file per column), the trailing slash on the path given to get changes when the columns are mapped:

`:tab/ set ([] a:til 1000000; b:1000000?1.0)    / a small splayed table to try this on
t:get`:tab       / immediate: every column is mapped now, and stays mapped
t:get`:tab/      / deferred: columns are mapped when a query uses them, then unmapped

You can see the difference in .Q.w[]mmap. In our test with a two-column table, it was 16 MB straight after the immediate get. After the deferred getit was 0, and it was still 0 after aselect`, because the columns were unmapped again once the query finished.

• Immediate mapping pays the cost of mapping once. It suits data that will be queried again and again.
• Deferred mapping avoids mapping columns that are never used, but pays the cost again on each query.

.Q.MAP

An HDB loaded with \l maps partition files on demand during each query. .Q.MAP[] maps them all once, up front, and keeps them mapped, so queries don't pay for opening and mapping the same files again and again. KX's notes on it:

• It needs more file handles and more file maps from the OS. On Linux, you may need to raise the vm.max_map_count limit.
• It is not recommended for compressed files, because the uncompressed maps stay in memory.
• It doesn't work with linked columns or with virtual partition columns.

Threads or Processes?

To query an HDB in parallel, peach (parallel each) can share out work in two ways:

• Secondary threads, q -s 4: one q process, with 4 extra threads inside it.
• Secondary processes, q -s -4: the main q process sends the work to separate q processes. You start those yourself and list their handles in .z.pd.

The book Q Tips (Nick Psaris, 2015) points out that the difference between the two is subtle, mostly because of memory mapping. The diagram shows why:

Memory: Closer Than You'd Think

You might expect five processes to need five times the memory of one. For the data in an uncompressed HDB, they don't:

• Mapped files are shared. Every process that maps a column file reads the same pages in the page cache, as covered above. Five processes reading the same price file use one copy of it in RAM.
• Each process (and each thread) still has its own heap for query results and temporary lists. KX notes that each secondary thread's heap is at least 64 MB.
• .Q.MAP[] in each process maps the partitions once, up front, so the processes don't re-map files on every query.

That is why KX describes the motivating use case for secondary processes as "multiprocess HDBs, combined with non-compressed data and .Q.MAP[]".

Compressed files break the sharing. Compressed data has to be decompressed before q can use it, and KX explains that decompressed data has no file behind it, so kdb+ reserves memory for it in the process doing the reading. Several processes reading the same compressed column can each end up holding their own decompressed copy. This is also why KX doesn't recommend .Q.MAP for compressed files: the decompressed data then stays in memory.

Other Differences (from KX's Documentation)

Key Takeaways

• Mapping a file gives it addresses. It doesn't load it.
• Pages arrive on first touch (page faults), and live in the page cache, shared by every process.
• Mapped pages count towards a process's RSS once touched, but they aren't part of q's heap.
• Query results and temporary lists are new objects in q's heap.
• Read less: filter on the partition column and select only the columns you need.

Mapping works naturally for columns whose values are all the same size, such as longs, floats and timestamps. Columns of lists, such as strings, need extra machinery, covered in Anymap.