The Four Attributes

What Are Attributes?

Attributes are metadata tags that you apply to a list or a table column to tell kdb+ about its structure. kdb+ uses this information to choose a faster algorithm — instead of scanning every element to answer a query, it can use an index or exploit the sorted order of the data.

Descriptive, not prescriptive. An attribute is an assertion that the data already has a certain structure. It does not sort or reorganise the data for you.

Three of the attributes validate the data when you apply them:

• s# — fails with 's-fail if the list is not in ascending order
• u# — fails with 'u-fail if the list contains duplicates
• p# — fails with 'u-fail if equal values are not contiguous (e.g. `p#1 1 2 3 4 1 2 fails because the 1s are not all together)
• g# — has no structural requirement, so it can always be applied

After an attribute is applied, kdb+ does not keep the data in that shape for you. When an operation changes the list, kdb+ either keeps the attribute (if it can be sure it still holds) or silently removes it — it does not signal an error. Check with attr after changes if the attribute matters.

There are four attributes:

Sorted — `s#

The sorted attribute tells kdb+ that a list is in ascending order, so searches can use binary search instead of a linear scan:

q)L: `s# 1 2 3 5 8 13 21
q)attr L
`s
q)L bin 8
4
q)L ? 13
5

asc returns a list with the sorted attribute already applied. s# is lost if you append a value smaller than the current last value:

q)L: `s# 1 2 3
q)L,: 0
q)attr L
`

Unique — `u#

The unique attribute builds a hash table from each value to its position, so lookups no longer scan the list:

q)ids: `u# `AAPL`GOOG`MSFT
q)attr ids
`u
q)ids ? `GOOG
1

u# is kept on append as long as the new values are still unique. Appending a duplicate does not signal an error — the append succeeds and the attribute is silently removed. So u# speeds up lookups, but it does not enforce uniqueness.

It suits reference data where each value appears exactly once: instrument identifiers, currency codes, the key of a lookup table.

Parted — `p#

The parted attribute tells kdb+ that equal values appear in contiguous blocks (all AAPL rows together, all GOOG rows together, etc.). kdb+ records where each block starts:

q)syms: `p# `AAPL`AAPL`AAPL`GOOG`GOOG`MSFT
q)attr syms
`p
q)syms ? `GOOG
3

For a filter like where sym=`AAPL, kdb+ reads only the AAPL block. This makes p# the standard attribute for the sym column in each partition of a date-partitioned HDB, where the data is sorted by sym when it is written at end of day.

p# is removed by any operation that modifies the list in memory — including an append, even one that would keep the values contiguous:

q)L: `p# `AAPL`AAPL`GOOG`GOOG
q)L,: `GOOG
q)attr L
`

Grouped — `g#

The grouped attribute makes kdb+ build and maintain an index from each distinct value to all of its positions — the same structure that group returns:

q)syms: `g# `AAPL`GOOG`AAPL`MSFT`GOOG
q)attr syms
`g
q)group syms
AAPL| 0 2
GOOG| 1 4
MSFT| ,3

Grouped does not require the data to be sorted or contiguous, and the index is updated as values are appended in any order. This makes it the usual choice for the sym column in the RDB, where new data arrives continuously in unpredictable order.

The cost of g# is the memory for the index and a little extra work on each append.

Memory Overhead

KX does not publish exact per-attribute memory formulas, but the shape of the cost follows from what each attribute stores:

As a rough illustration, measured with .Q.w[] on KDB-X: for a 10-million-row symbol column with 5,000 distinct values, p# added very little memory, while g# added roughly 90MB. For a 10-million-row column of unique longs, u# added roughly 400MB. (These figures include the memory allocator's rounding to powers of two, so treat them as orders of magnitude.) You can measure it yourself like this:

x: asc 10000000?`$string til 5000
before: .Q.w[]`used
y: `g#x                      / an attributed copy of x
(.Q.w[]`used) - before       / the copy plus the index

Attribute Preservation Rules

What happens to each attribute when an in-memory list changes:

On disk, appending to a column file keeps only s#; the other attributes must be re-applied after an append. In practice, the HDB applies p# at end of day to sorted data that is not appended to afterwards, and the RDB uses g# because it is kept through the continuous stream of appends.