Broad backwards- & forwards-compatability test

[wjones127]

We want broader testing that datasets written with old version of libraries are readable with new libraries.

Coverage

For wide coverage, we can have one set of data that we write. It should have most possible data types (all primitives plus common nested combinations) and all index types (with some parameter variations).

We should also vary:

• Lance file version (don't need to care about unstable versions)
• Lance index versions
• Field compression settings

Forwards compat testing

See:

Backwards compat

• Scalar index types
  • JSON index
  • N Gram
  • Zone map
  • BTree
  • Bitmap
  • IVF-PQ
    • Old index versions
  • IVF_HNSW_PQ
  • IVF HNSW SQ
  • FTS
  • Label list
  • Bloom filter

Tests

Forwards compat

• Full scan
• Query the index
• Test optimize indices
• Test compaction
• Test appending

Backwards compat

• Full scan
• Query the index
• Test optimize indices
• Test compaction
• Test appending