Attributes in the Tick Architecture

Where Each Attribute Lives

The main processes in a kdb+ tick system use different attributes, for reasons that follow directly from how each process handles data:

How the RDB Gets g# on sym

In KX's kdb+tick, the tickerplant's .u.tick function applies the grouped attribute to the sym column of every table in its schema. When the RDB subscribes, the tickerplant returns each table's empty schema — including that attribute — and the RDB (tick/r.q) creates its tables from it:

/ From tick/r.q: initialise the schema, then replay the tickerplant log
.u.rep:{(.[;();:;].)each x;if[null first y;:()];-11!y;system "cd ",1_-10_string first reverse y};

(.[;();:;].)each x assigns each (tableName;emptyTable) pair, so the RDB's tables start out with g# on sym. Because g# is kept on append, it survives the log replay and every update that follows.

At end of day, r.q saves the tables to the HDB with .Q.hdpf, which empties them, and then puts g# back on sym for the next day:

/ From tick/r.q
.u.end:{t:tables`.;t@:where `g=attr each t@\:`sym;.Q.hdpf[`$":",.u.x 1;`:.;x;`sym];@[;`sym;`g#] each t;};

You can also apply it yourself to an in-memory table:

trade: ([] sym:`AAPL`GOOG`AAPL; price:185.2 142.8 185.5)
update `g#sym from `trade        / or: @[`trade;`sym;`g#]
attr trade`sym                    / `g

With g# in place, a query like select from trade where sym=`AAPL goes straight to AAPL's list of positions instead of scanning the whole table.

Why Not p# on the RDB?

p# requires equal values to be contiguous, and it is removed by any in-memory append. In a live market data stream, rows for different symbols are interleaved — AAPL, GOOG, AAPL, MSFT … — so p# could not even be applied, let alone kept. Keeping the table sorted by sym would mean re-sorting it on every update. g# has no ordering requirement and is kept on every append.

p# in the HDB

At end of day, the data is sorted by sym as it is written to each date partition, so all rows for AAPL are together, all rows for GOOG are together, and so on. .Q.hdpf and .Q.dpft do this for you: they sort each table by the column you name and apply p# to it on disk. Done by hand for one partition, it looks like this:

`sym xasc `:/data/hdb/2025.01.01/trade/       / sort the splayed table on disk by sym
@[`:/data/hdb/2025.01.01/trade/; `sym; `p#]   / then apply parted to the sym column

When a query runs where sym=`AAPL against that partition, kdb+ reads only the rows of the AAPL block. For a partition with 50 million rows and 5,000 instruments, that is on average about 10,000 rows rather than 50 million.

Attributes for As-Of Joins

aj[symtime; trade; quote] finds, for each trade, the most recent quote for the same sym at or before the trade's time. KX's reference for aj recommends:

• In memory: the grouped attribute on sym in the quote table, with time sorted within each sym
• On disk: the parted attribute on sym, with time sorted within each sym

quote: `sym`time xasc quote        / time sorted within sym
update `g#sym from `quote          / grouped attribute on sym
aj[`sym`time; trade; quote]

The sorted attribute is not required on time — and after sorting by sym then time, the time column as a whole is usually not in ascending order, so s# could not be applied to it anyway. KX notes that departing from this layout carries a severe performance penalty, and that on disk g# does not help.

u# for Reference Data

An instrument master table — mapping each sym to its name, exchange, currency and other static fields — is a natural fit for u# on its key:

instMaster: ([sym:`u#`AAPL`GOOG`MSFT] name:("Apple";"Google";"Microsoft"))
instMaster`GOOG                   / hash lookup on the key

Note that u# does not itself prevent duplicates: appending a duplicate to a u# list simply removes the attribute. Here, the table being keyed is what stops a second AAPL row from being inserted (insert signals 'insert for an existing key).