Serialization

Overview

Serializing a value turns it into a sequence of bytes that can be sent to another process or stored, and deserializing turns those bytes back into the value. kdb+ does this every time processes talk over IPC: the sender serializes the message, and the receiver deserializes it. q also lets you call these steps yourself:

Function	What it does
`-8!x`	Serialize `x` to its IPC bytes
`-9!x`	Deserialize IPC bytes back to a value
`-18!x`	Serialize `x`, compressing it if that pays off
`-22!x`	The size of `x` serialized, without building the bytes

Serializing and Deserializing

-8! returns a byte vector (type 4h), and -9! turns it back into the original value:

-8!1 2 3

0x0100000026000000070003000000010000000000000002000000000000000300000000000000

-9!-8!1 2 3    / 1 2 3

Almost any q value can be serialized, including tables, dictionaries and functions:

t:([] sym:`AAPL`MSFT; price:185.2 410.1)
b:-8!t
type b       / 4h
t~-9!b       / 1b — a perfect round trip

f:{x+1}
(-9!-8!f) 5  / 6

Bytes that aren't a valid message signal 'badmsg.

Reading the Bytes

Every serialized message starts with an 8-byte header, followed by the data. Here is an int atom, 1i:

-8!1i

0x010000000d000000fa01000000

Bytes	Meaning
`01`	The data is little-endian (`00` would mean big-endian)
`00`	The message type: `0` async, `1` sync, `2` a response
`00 00`	Not needed for this example
`0d000000`	The length of the whole message, 13 bytes, as a little-endian int
`fa`	The type of the data: `0xfa` is -6 as a signed byte, the type of an int atom
`01000000`	The value, 1, as a 4-byte little-endian int

A list adds an attribute byte and a 4-byte count before its items. In -8!1 2 3 above, 07 is the type (a long list), 00 means no attribute, 03000000 is the count, and each long then takes 8 bytes.

You will rarely need to decode bytes by hand. It helps to know the layout when you write a client in another language, or debug a connection with a packet capture.

Size and Compression

-22!x gives the size of x serialized. It is the same number as count -8!x, but quicker, because it doesn't build the bytes:

-22!til 1000             / 8014

-18!x is like -8!, but compresses the result. It only compresses when both of these are true:

the uncompressed bytes are longer than 2000 bytes, and
compressing makes the data less than half its original size.

Otherwise -18! returns the same bytes as -8!. -9! reads both forms:

count -18!til 1000           / 3276 — compressed
(-8!1 2 3)~-18!1 2 3         / 1b — too small to compress
til[1000]~-9!-18!til 1000    / 1b

Compression over IPC

kdb+ applies the same rules to real IPC traffic. Since 2012, kdb+ compresses an outgoing message automatically when:

the serialized message is longer than 2000 bytes,
the connection is not to localhost or 127.0.0.1, does not use a Unix domain socket, and (since kdb+ 4.1t 2021.06.04) does not otherwise resolve to the local machine, and
compressing makes the data less than half its original size.

So a process sending to another process on the same machine never compresses, and you don't need to do anything to turn compression on across machines.

Talking to Other Versions

When a client connects, it sends a capability byte as part of the handshake, and the server replies with the capability they have in common. That is how processes on different kdb+ versions agree on what the connection can do, such as compression, or messages larger than 2GB.

Saving to Disk

To save a value to a file and load it back, use set and get. You don't need to serialize it yourself:

`:tdata set t     / writes the table t to the file tdata
get `:tdata       / reads it back

For large tables, use the splayed and partitioned layouts covered in the Splayed Tables and HDB topics, so that queries read only the columns they need.