Detect How a File Was Written, and Why It Matters for Large Files

Question

Given an on-disk object, how can you determine whether it was written using:

set

or

1:

Why does this distinction matter when working with very large files?

Solution

The easiest way to check is to inspect the type returned by get:

type each get`:tab

Typical results:

0 0h

or

77 77h

Interpretation:

Files written with 1: remain individually memory-mappable, making them suitable for working with datasets that are much larger than available RAM.

Example

Written with set

`:file1 set ((1;`a);(2;`b))

type each get`:file1

Result:

0 0h

Written with 1:

`:file2 1: ((1;`a);(2;`b))

type each get`:file2

Result:

77 77h

Explanation

What Does set Produce?

When an object is written using:

`:file set x

q serializes the object as an ordinary list.

Later:

get`:file

must reconstruct that list in memory.

Conceptually:

Disk
  ↓
Deserialize
  ↓
RAM

The entire object must be materialized before it can be used.

What Does 1: Produce?

Writing with:

`:file 1: x

creates an Anymap.

Anymaps were designed specifically for large objects that may not fit comfortably in memory.

Instead of requiring the entire object to be loaded:

Disk
  ↓
Memory Mapping
  ↓
Access only required regions

The operating system can map portions of the file on demand.

This is why type each get returns:

77h

The type tag identifies the object as an Anymap rather than a normal list.

Why This Matters

For small files there is little difference:

10 KB
100 KB
1 MB

Everything fits comfortably in RAM.

For very large files:

100 GB
500 GB
2 TB

the distinction becomes significant.

An ordinary serialized object may require the entire structure to be reconstructed in memory.

An Anymap allows the operating system to load only the pages that are actually accessed.

Working with Huge Files

Suppose a file contains several terabytes of data:

File Size: 2 TB
RAM:       64 GB

Loading the entire object would be impractical.

Instead, the preferred approach is:

Map
 ↓
Read only required rows
 ↓
Process
 ↓
Release mapping

For example:

x:get`:hugefile

followed by a query that accesses only a subset of the data.

The operating system will fault in only the pages required by that query.

A Practical Guideline

When working with extremely large mapped files:

Avoid:

Map entire file
       ↓
Keep mapping alive indefinitely

unless the file will be queried repeatedly.

Instead:

Map shortly before use
       ↓
Read required rows
       ↓
Allow mapping to be released

This keeps resource usage predictable and avoids maintaining large mappings that may never be accessed again.

Why This Is Useful

Anymaps are particularly valuable when:

• Historical datasets are larger than RAM
• Only a small fraction of the file is accessed at any one time
• Many large files exist, but only a subset are queried
• You want the operating system to manage page loading automatically

This allows kdb+ systems to work efficiently with datasets that are hundreds of gigabytes or even terabytes in size.

Key Takeaway

Use:

type each get`:file

to determine how an object was written.

0h   -> ordinary serialized object (set)
77h  -> Anymap (1:)

Objects written with 1: remain individually memory-mappable and are therefore much better suited to very large datasets where only a small portion of the data needs to be accessed at any given time.