Immediate vs Deferred Memory Mapping

Question

When loading an on-disk object with get, what is the difference between:

get`:tab

and

get`:tab/

When would you choose one over the other?

Solution

get`:tab     / immediate mapping
get`:tab/    / deferred mapping

Here tab is a splayed table — a directory with one file per column, written with something like ```:tab/ set t``. Both forms use memory mapping rather than copying the columns into RAM.

The difference is when the operating system mapping is created.

Immediate mapping is generally preferred when the data will be accessed repeatedly.

Deferred mapping is useful when many large objects may be loaded, but only a subset will actually be queried.

Explanation

What Is Memory Mapping?

Memory mapping allows a file on disk to be accessed through virtual memory rather than being copied into the process heap.

Instead of:

Disk File
     ↓
Copy Entire File
     ↓
Process Memory

the operating system creates a mapping:

Disk File
     ↓
Virtual Address Space
     ↓
Pages loaded on demand

This allows files much larger than available RAM to be accessed efficiently.

Immediate Mapping

Loading a file without a trailing slash performs an immediate mapping:

immediate:get`:tab

The mapping is created immediately and remains active until the variable is cleared or reassigned.

You can observe this using .Q.w[]:

.Q.w[]

Example:

used| 423104
mmap| 1600032

The increase in mmap indicates that the file has been mapped into the process address space.

Subsequent accesses reuse the same mapping:

select from immediate
select from immediate
select from immediate

No additional mapping work is required.

Deferred Mapping

Appending a trailing slash creates a deferred mapping:

deferred:get`:tab/

Immediately afterwards:

.Q.w[]

may show:

used| 421776
mmap| 0

No mapping has been created yet.

Instead, q stores a lightweight descriptor indicating where the data lives on disk.

Only when the object is actually accessed does q request a mapping from the operating system:

deferred

At that point the file is mapped and accessed.

Under the Hood

The behaviour can be observed using Linux strace.

Immediate mapping:

a:get`:tbl
select a from a
select a from a

Produces something similar to:

mmap(...)
select
select
munmap(...)

The mapping is created once and reused.

Deferred mapping:

a:get`:tbl/
select a from a
select a from a

Produces something closer to:

mmap(...)
query
munmap(...)

mmap(...)
query
munmap(...)

Each access creates a temporary mapping.

This behaviour has been demonstrated by KX engineers using Linux strace, which exposes the underlying mmap and munmap system calls.

Reproducing This Yourself with strace

On Linux, the system calls used by a running q process can be observed using strace.

Start q normally:

q

In a second terminal, find the process id:

ps -ef | grep q

Example:

jeremy   12345 ...

Attach strace to the running process:

sudo strace -p 12345

To focus specifically on memory mapping operations:

sudo strace -e mmap,munmap -p 12345

Now return to q and run:

a:get`:tbl

You should observe an mmap(...) system call.

Then:

a:()

should produce a corresponding:

munmap(...)

For deferred mapping:

a:get`:tbl/

you should see no mapping activity initially.

However, when the data is accessed:

select from a

you should observe:

mmap(...)
...
munmap(...)

This provides a useful way to verify exactly when kdb+ is creating and releasing mappings on your platform.

Important: Mapping Is Not The Same As Reading From Disk

A common misunderstanding is:

mmap
   ↓
Read Entire File

This is not what happens.

Even after a mapping exists:

File
  ↓
Virtual Address Mapping

the file's pages are still loaded only when accessed.

Likewise, repeated mapping does not necessarily imply repeated disk reads.

The operating system maintains a page cache:

Disk
  ↓
Page Cache
  ↓
Process

If a page is already cached, the operating system may satisfy the access entirely from memory.

Therefore:

Repeated mmap calls

does not automatically mean:

Repeated physical I/O

Performance Implications

Suppose both objects contain identical data:

immediate:get`:tab
deferred:get`:tab/

(select from immediate)~select from deferred

Result:

1b

However:

\t:10 select from immediate

might produce:

0

while:

\t:10 select from deferred

might produce:

22

The difference comes from the repeated mapping overhead associated with deferred access.

Relationship to .Q.MAP

The purpose of .Q.MAP becomes easier to understand once the difference between immediate and deferred mapping is clear.

Without .Q.MAP:

Query 1
  mmap
  query
  munmap

Query 2
  mmap
  query
  munmap

Query 3
  mmap
  query
  munmap

With .Q.MAP:

Startup
  mmap

Query 1
  query

Query 2
  query

Query 3
  query

.Q.MAP[] forces HDB partitions to be mapped immediately and remain mapped.

This removes repeated mapping overhead and can significantly improve query performance.

However:

• It increases the amount of virtual address space in use
• It increases the number of open file handles
• It should be used carefully with compressed databases

Choosing Between Immediate and Deferred Mapping

Immediate mapping is usually preferred when:

• The data will be queried repeatedly
• The object forms part of an active workflow
• Minimising query latency is important

Examples:

trade:get`:trade
quote:get`:quote

Deferred mapping is useful when:

• Many large objects may be loaded
• Only a subset will actually be queried
• You want to avoid maintaining mappings for data that may never be touched

Examples:

archive:get`:archive/
backup:get`:backup/

Key Takeaway

Both:

get`:tab

and

get`:tab/

use memory mapping rather than copying data into RAM.

The difference is simply when the mapping is created:

get`:tab
     ↓
Map now
     ↓
Reuse mapping

get`:tab/
     ↓
Map only when needed
     ↓
Potentially remap on future accesses

Immediate mapping is generally best for frequently accessed data.

Deferred mapping is useful when loading large datasets speculatively, where some objects may never actually be used.

When repeatedly querying an HDB, consider .Q.MAP[], which keeps partitions mapped and can significantly reduce mapping overhead.