Compress Existing Partitions Column by Column

Question

The partitions were written without compression. Write a script to compress every column in every date partition using gzip at level 6 with a logical block size of 17.

Assume path and dts are already defined.

Solution

compressPerDate:{[d]
  tblPath:.Q.par[path;d;`t];
  columns:(key tblPath) except `.d;
  {[p;c]
    colPath:.Q.dd[p;c];
    (colPath;17;2;6) set get colPath
   }[tblPath;] each columns
 }
compressPerDate each dts

Verify afterwards:

-21! `:/hdb/2025.01.15/t/price

compressedLength  | 2082
uncompressedLength| 2728
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 6i

Explanation

key tblPath returns all files inside the table directory as a list of symbols. This includes every column file and the .d metadata file.

except `.d removes .d from the list. The .d file records the column order and is not a data column, so it is left as it is.

.Q.dd[tblPath; colName] constructs the full path to a column file by appending the column name to the table path:

.Q.dd[`:/hdb/2025.01.15/t; `price]
/ → `:/hdb/2025.01.15/t/price

(colPath; 17; 2; 6) set get colPath is the compressed write pattern:

• get colPath — reads the column data into memory
• (path; logicalBlockSize; algorithm; zipLevel) set data — writes the data back to the same path with compression

The four-element LHS of set is the compressed file descriptor:

• 17 — logical block size (2^17 = 128 KB blocks)
• 2 — algorithm (2 = gzip)
• 6 — zip level (1–9; higher = smaller file, slower to compress)

This reads and rewrites each column file in place. The original uncompressed file is replaced by the compressed version with the same name. Because .z.zd is not set, this approach gives you per-column control — useful when some columns (like a random float column) compress poorly and should be left uncompressed.

To apply the same settings globally for all writes in the current process instead, set .z.zd before writing:

.z.zd: 17 2 6   / all subsequent set/upsert calls use these compression settings