learnkdb
Back to Attributes in the Tick Architecture

Attributes · Attributes in the Tick Architecture · Practice Question

Why Does the RDB Use `g#` While the HDB Uses `p#` on sym?

Sign in to track your progress

Question

Both the RDB and HDB apply an attribute to the sym column, but they use different ones — g# in the RDB and p# in the HDB. Why?

Solution

/ RDB: grouped attribute — applied at startup, maintained on all appends
@[`trade; `sym; `g#]

/ Confirm it survives continuous updates
/ (subscribe and receive 1000 updates...)
attr trade`sym    / still `g — maintained automatically

/ HDB: parted attribute — applied after the EOD sort
/ (.Q.hdpf / .Q.dpft do this for you; by hand it is:)
`sym xasc `:/data/hdb/2025.01.01/trade/
@[`:/data/hdb/2025.01.01/trade/; `sym; `p#]

/ In the HDB process, after loading the database:
meta trade                                   / sym shows p in the a column

Explanation

The choice comes down to one question: can the data be kept sorted by sym?

In the RDB, the answer is no. Market data arrives from the tickerplant in the order it was generated by the exchange — a continuous stream where AAPL, GOOG, AAPL, MSFT, AAPL can all arrive within a millisecond. Maintaining sorted order would require inserting each row in its correct position (O(n) insert cost) or re-sorting the entire table on every update (O(n log n)).

p# would be silently dropped on the very first append where a symbol's order changes (e.g. an AAPL row arriving after a GOOG row that already ended the AAPL block). Applying p# to the RDB is simply not viable.

g# builds an inverted index. The RDB's sym column looks like AAPL GOOG AAPL MSFT GOOG... with no particular ordering — and g# works perfectly on this. The index is updated on every insert. A where sym=`AAPL filter goes directly to AAPL's position list without scanning any other rows.

In the HDB, the answer is yes. At EOD, before writing to disk, the data is sorted by sym with xasc. After sorting, all AAPL rows are contiguous, all GOOG rows are contiguous. This is the exact structural guarantee that p# requires. Applying p# to sorted, contiguous data creates a compact block index with very low memory overhead.

For HDB queries, p# is strictly better than g# because:

  • p# uses less memory (one entry per distinct value; g# stores all row positions)
  • p# enables contiguous disk reads (the AAPL block is physically adjacent on disk)
  • g# is designed for unsorted in-memory data — KX notes, for example, that on disk the grouped attribute does not help aj