Why Mapping a Huge File Doesn't Use a Huge Amount of RAM

Question

A 500GB file gets memory-mapped in a process running on a machine with only 16GB of RAM.

Why doesn't this immediately fail, or consume anywhere near 500GB of memory?

How can kdb+ query data from a file that is much larger than the available RAM?

Solution

Memory mapping does not load a file into RAM.

Instead, the operating system creates a mapping between a range of the process's virtual address space and the file on disk.

For example:

t:get`:huge_table/

When this executes, q only sets up the mapping. No file data is read and almost no physical RAM is consumed.

Physical memory is only used later, one page at a time, as portions of the file are actually accessed.

Explanation

Virtual Memory vs Physical Memory

A common misconception is:

Map File
     ↓
Load Entire File Into RAM

What actually happens is:

Map File
     ↓
Create Virtual Address Mapping
     ↓
Load Pages On Demand

Every process operates using virtual addresses, not physical RAM addresses.

When a file is mapped, the operating system reserves a range of virtual addresses and associates them with locations inside the file.

For example:

File Offsets                     Virtual Addresses

0 ---------------------------> 0x1000000000
4KB -------------------------> 0x1000010000
8KB -------------------------> 0x1000020000
...
500GB -----------------------> 0x8d00000000

The operating system has effectively created a lookup table:

Virtual Address
       ↕
File Offset

No file contents have been loaded into memory.

At this stage:

File Size:          500 GB
Physical RAM Used:  ~0 MB

The operating system has simply recorded where data should come from if those addresses are ever accessed.

Why Doesn't Mapping a 500GB File Require 500GB of RAM?

The virtual address range must be large enough to represent the entire file, but virtual memory is not physical memory.

On a modern 64-bit machine, the available virtual address space is enormous (typically hundreds of terabytes).

Reserving 500GB of virtual addresses is therefore trivial.

Think of it like reserving house numbers on a street:

House numbers reserved:
1 ... 500,000,000

Actual houses built:
0

The house numbers themselves consume almost no space.

Similarly, virtual addresses consume very little memory until they are actually backed by physical RAM.

What Happens When Data Is Accessed?

Suppose a query touches the first value of a column:

first t[`price]

The CPU attempts to read a virtual address belonging to the mapped file.

The operating system checks:

Is this virtual address currently backed by RAM?

If not, a page fault occurs.

CPU reads virtual address
          │
          ▼
      Page Fault
          │
          ▼
OS identifies file offset
          │
          ▼
Required page read from disk
          │
          ▼
Page loaded into RAM
          │
          ▼
Instruction resumes

The process continues as though it was reading ordinary memory.

What Is a Page?

The operating system manages memory in fixed-size blocks called pages.

Typically:

Page Size ≈ 4 KB

When data is accessed, the operating system loads the page containing it. (In practice it usually also reads a little ahead, anticipating that nearby pages will be needed next — but it never reads the whole file just because one value was touched.)

For example:

500 GB File

+----------------------------------------------------+
|[####]                                              |
+----------------------------------------------------+

Only the highlighted page might be loaded:

RAM

+------+
| 4 KB |
+------+

The remaining 499.999+ GB remains on disk.

What Happens During a Large Scan?

Suppose the price column is 100GB and a query needs to scan the entire column:

select max price from trade

Conceptually:

Load Page 1
Scan Page 1

Load Page 2
Scan Page 2

Load Page 3
Scan Page 3

...

Load Page N
Scan Page N

The query gradually streams through the file.

kdb+ simply reads memory addresses.

The operating system automatically loads pages as they are needed.

How Does the OS Know Which Pages to Load?

The query itself determines the access pattern.

For example:

select from trade where sym=`AAPL

First, kdb+ begins reading the sym column.

As it touches addresses belonging to the sym file:

trade/sym

the corresponding pages are faulted into RAM.

If the query later needs:

trade/price

then pages from the price column are loaded.

The operating system does not know anything about symbols, prices, or queries.

It only sees memory accesses and loads the pages required to satisfy them.

Is Mapping the Slow Part?

No.

Creating the mapping is usually extremely fast because no data is read.

Map 500 GB file
≈ milliseconds

The expensive part is touching pages for the first time because this requires disk I/O.

This is why:

get`:trade/

often appears almost instantaneous, while the first query against the table may take longer as pages are loaded into RAM.

Why Is This Important For kdb+?

This behaviour is one of the key reasons kdb+ can efficiently work with historical databases that are far larger than available RAM.

Consider:

trade table
-----------
500 GB

A query such as:

select avg price by sym from trade where date=2025.01.15, sym=`AAPL

typically touches:

One partition
A subset of rows
A handful of columns

Only the pages containing that data are loaded into RAM.

500 GB File
      │
      ▼
Touch 20 MB
      │
      ▼
Load ~20 MB

The remainder of the file stays on disk.

This allows a machine with 16GB of RAM to efficiently query databases that are hundreds of gigabytes or even terabytes in size.

Key Takeaway

Memory mapping does not load a file into RAM.

It creates a mapping between a file on disk and a region of virtual address space. Physical memory is consumed only when pages are accessed, and only for the portions of the file that are actually touched.

This demand-paging mechanism is what allows kdb+ to work efficiently with datasets that are far larger than the available RAM.