Right now, each leaf node in the tree keeps a single buffer and flushes all elements at once. This consumes much more memory than is necessary. Instead, allocate buffers of fixed sizes (eg: 64k) and flush to a "window" whenever the buffer is full.
Right now, each leaf node in the tree keeps a single buffer and flushes all elements at once. This consumes much more memory than is necessary.
Instead, allocate buffers of fixed sizes (eg: 64k) and flush to a "window" whenever the buffer is full.