Sort In-Memory Before Saving with Parted Attribute

Question

Instead of sorting the data on disk after persisting, sort each date slice in memory before writing and apply the p# attribute to the sym column. Rewrite the persistPerDate function to do this in a single step.

Solution

N:1000
t:([] date:N?.z.d + til 3;
      time:N?100?23:59:59.999;
      sym:N?`A`B`C;
      price:N?10.0;
      qty:N?1000j)

path:`:/hdb
dts:exec distinct date from t

persistPerDate:{[d]
  data:update `p#sym from `sym`time xasc delete date from select from t where date = d;
  tblPath:` sv path,`$string d,`t,`;
  tblPath set .Q.en[path;data]
 }
persistPerDate each dts

Explanation

The key change is the construction of data:

data:update `p#sym from `sym`time xasc delete date from select from t where date = d

Reading right-to-left through the functional chain:

1. select from t where date = d — filter to today's rows
2. delete date from ... — remove the date column (it belongs in the path, not the data)
3. `sym`time xasc ... — sort the result by sym, then time within each sym group
4. update `p#sym from ... — apply the parted attribute to the sym column on the now-sorted table

When set writes this table to disk, it preserves the attribute. The written column file includes the p# metadata, so the partition is ready for fast sym-keyed queries immediately after the write — no post-save sort step is needed.

In-memory vs on-disk sort — which is faster?

Sorting in memory before writing is generally faster than sorting on disk after writing, because:

• The sort operates on compact in-memory arrays rather than reading and rewriting individual column files
• Only one write pass is needed (sort + write) vs two (write then rewrite via xasc)

The on-disk sort approach from the previous question is more useful when you are working with data that is already on disk (for example, re-sorting a partition after appending new rows), or when the in-memory slice is too large to sort comfortably in RAM.