Due to how ingested data is sorted and how compressed data is sorted both when writing to disk and when reconstructing the data in the query engine, issues related to the order of the data might happen.
One example of this is if out-of-order data points are added to the storage engine which then sorts and compresses them. This could cause overlapping compressed segments which could cause issues during querying. It is not certain that this causes issues but the problem should be researched to ensure this does not cause issues. A discussion on this was started here: #293 (comment).
Another potential issue is the disconnect between what Apache DataFusion expects from the ordering of the output from SortedJoinExec and what we currently provide. We currently only guarantee that the order is the same for each field column. It might be necessary to research what impact setting the sort guarantees and requirements has on the execution in Apache DataFusion and considering whether it would be better to remove the currently set guarantees. A discussion on this was started here: #293 (comment).
Due to how ingested data is sorted and how compressed data is sorted both when writing to disk and when reconstructing the data in the query engine, issues related to the order of the data might happen.
One example of this is if out-of-order data points are added to the storage engine which then sorts and compresses them. This could cause overlapping compressed segments which could cause issues during querying. It is not certain that this causes issues but the problem should be researched to ensure this does not cause issues. A discussion on this was started here: #293 (comment).
Another potential issue is the disconnect between what Apache DataFusion expects from the ordering of the output from
SortedJoinExecand what we currently provide. We currently only guarantee that the order is the same for each field column. It might be necessary to research what impact setting the sort guarantees and requirements has on the execution in Apache DataFusion and considering whether it would be better to remove the currently set guarantees. A discussion on this was started here: #293 (comment).