Reduce Memory Use by Selecting Only Needed Columns

Question

Querying a large memory-mapped HDB table, why does select price from t where date=x use noticeably less memory than select from t where date=x, even though both filter the same rows?

Solution

select price from t where date=x   / only `price` is touched
select from t where date=x         / every column is touched

Explanation

Columns on disk stay memory-mapped — and therefore cheap — until something actually reads them. Naming only the columns a query needs means only those columns get paged in for the matching rows; every other column is left alone, mapped but untouched.

select from t (every column) forces every column's matching rows into RAM, whether the query needs them or not. Multiply that by however many partitions or symbols a query spans, and selecting unnecessary columns can be the difference between a query that fits comfortably in memory and one that doesn't.

The same logic applies to the virtual column i (row index): referencing it forces materialisation of row numbers that would otherwise never need to exist as real data, so it's worth using only when actually needed — for example, when an aggregation genuinely depends on row position.