Unify types for managing storage

[skejserjensen]

This PR removes all types for managing storage except for modelardb_storage DataFolder. Thus, modelardb_embedded DataFolder, TableMetadataManager, MetadataManager, etc have been removed. In addition, the multiple instances of SessionContext that were used have been reduced to a single SessionContext that is part of modelardb_storage DataFolder. Thus, modelardb_storage DataFolder now represents a location that stores data and can query data, and all methods for operating on data at that location are part of it. An attempt to split modelardb_storage DataFolder into two types was made, where one type handles files and one provides abstractions on top of files in the form of tables. However, as the dependency between the instance variables and methods of these two types made implementing them very complex, and most operations are already on tables due to deltalake, it was decided to simply implement data storage and querying as modelardb_storage DataFolder. There is still some overlap between the methods in Context and modelardb_storage DataFolder, e.g., for registering tables, but removing the duplicate code seems to require significant changes due to the difference between how DataSink is used in modelardb_server and modelardb_embedded. Likewise, the methods for handling normal tables and metadata tables in modelardb_storage DataFolder can probably be reduced to one set of methods. However, since the purpose of this PR was to reduce the number of types for managing storage to one, the PR was already very big, and there are issues for reducing the number of methods for specific table types, is simplifying and combining methods purposely not part of this PR.

[Copilot]

Pull Request Overview

This pull request consolidates storage management types by removing redundant abstractions. The main change unifies all storage operations under modelardb_storage::DataFolder, eliminating separate types like modelardb_embedded::DataFolder, TableMetadataManager, and MetadataManager. The PR also reduces multiple SessionContext instances to a single one managed by DataFolder, simplifying the architecture while maintaining all existing functionality.

Key Changes:

• Removed DeltaLake and TableMetadataManager types; their functionality now resides in DataFolder
• Consolidated all session contexts into a single instance within DataFolder
• Metadata tables now use a metadata schema prefix for clearer organization

Reviewed Changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/modelardb_storage/src/data_folder.rs	New unified DataFolder implementation combining Delta Lake and metadata management
crates/modelardb_storage/src/delta_lake.rs	Removed (functionality moved to DataFolder)
crates/modelardb_storage/src/metadata/table_metadata_manager.rs	Removed (functionality moved to DataFolder)
crates/modelardb_storage/src/metadata/mod.rs	Removed
crates/modelardb_storage/src/lib.rs	Updated module exports and added metadata schema registration
crates/modelardb_server/src/storage/uncompressed_data_manager.rs	Updated to use unified DataFolder API
crates/modelardb_server/src/context.rs	Removed dedicated SessionContext; now accessed via DataFolder
crates/modelardb_manager/src/metadata.rs	Converted to trait-based approach with ManagerMetadata
crates/modelardb_embedded/src/operations/data_folder.rs	Removed embedded DataFolder implementation
crates/modelardb_compression/src/compression.rs	Added multivariate compression functions
crates/modelardb_compression/src/lib.rs	Updated public API exports

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.