Why a Large Join Can Spike Memory Above Either Input

Question

Joining two 10-million-row lists together briefly uses noticeably more memory than either list's own footprint, even though the inputs themselves haven't grown. Why, and how can that extra memory be reclaimed once the join is done?

Solution

a:til 10000000
b:til 10000000
\ts c:a,b            / allocates a new 20M-item list: about 268MB
.Q.gc[]              / returns 0 — a and b are still referenced, so nothing can be returned yet

delete a, b from `.  / once the inputs are no longer needed...
.Q.gc[]              / ...their memory can be returned to the OS

Explanation

, (join) builds an entirely new list holding both inputs' data — it doesn't reuse either input's memory, since a and b still exist independently afterwards. So while the join runs and until the inputs are released, q needs memory for a, b and the combined result at the same time. In this example, \ts reports about 268MB allocated for c (a 20-million-item list of longs, rounded up to its allocator size class), on top of the roughly 134MB each for a and b.

That combined footprint, not the eventual steady-state size of c alone, is often the real constraint on whether a large join fits in memory.

The join itself leaves no temporary garbage behind — the extra memory is simply the new result plus the inputs that are still alive. Calling .Q.gc[] straight after the join therefore returns nothing. To bring the footprint down, release the inputs once they're no longer needed (delete them, or reassign the variables). Their memory then returns to q's heap, and .Q.gc[] can return it to the operating system, as covered in the Garbage Collection section.