Optimize usage of tag columns in data during ingestion and querying

[CGodiksen]

Since tags are now stored in the data instead of in the metadata Delta Lake, the process for both ingestion and querying can be optimized to utilize this new structure. Specifically querying can be heavily optimized since we currently reconstruct the tag columns in each field column and then only use the tag columns from the first field column. We also reconstruct all tag columns and not only the requested tag columns.

We should look into ways of optimizing this process so we take advantage of the tags being available in the compressed segments and reconstruct the tag columns in an efficient manner. This could be achieved by creating a more complex plan of what is needed from each grid operator instead of always reconstructing as much as possible.

[CGodiksen]

We could consider modifying modelardb_compression::try_compress() so that it no longer requires a compressed schema specific to a model table and tag values but instead just creates a record batch with the base COMPRESSED_SCHEMA fields. Another function can then extend this record batch with the model table specific tag columns.

[CGodiksen]

We can consider pruning the Apache Parquet files in new_apache_parquet_exec using metadata and maybe_parquet_filters. In the same function we can also provide FileScanConfig with more information about the timestamps and values through Statistics.

We could also maybe use DeltaScan to automatically prune by time, value, and tags at the file and segment level. A discussion on this was started here: #293 (comment).

Generally pruning at the file and segment level using timestamp, value, and tags should be a focus even if it is not possible to do automatically using DeltaScan.

[skejserjensen]

We could consider modifying modelardb_compression::try_compress() so that it no longer requires a compressed schema specific to a model table and tag values but instead just creates a record batch with the base COMPRESSED_SCHEMA fields. Another function can then extend this record batch with the model table specific tag columns.

In general, I think it would make the compression API simpler by splitting it into steps that then can be combined to create a pipeline compressing multiple multivariate, one multivariate, multiple univariate, or one univariate time series. For example, if #297 is implemented, the steps could be something like sorting, splitting to multivariate, splitting to univariate, compressing, appending tags. A few helper methods should probably also be included that combine the steps into compression pipelines with commonly used configurations.

[skejserjensen]

Another optimization could also be to only write Delta Lake metadata for time series tables for the columns being used for filtering during query processing