feat: enhance repdef utilities to handle empty / null lists

[westonpace]

Empty & null lists are interesting. If you have them then your final repetition & definition buffers will have more items than you have in your flattened array. This fact required a considerably reworking in how we build and unravel rep/def buffers.

When building we record the position of the specials and then, when we serialize into rep/def buffers, we insert these special values. When unraveling we need to deal with the fact that certain rep/def values are "invisible" to the current context in which we are unraveling.

In addition, we now need to start keeping track of the structure of each layer of repetition in the page metadata. This helps us understand the meaning behind different definition levels later when we are unraveling.

This PR adds the changes to the rep/def utilities. We still aren't actually using repetition levels at all yet. That will come in future PRs.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 91.07692% with 87 lines in your changes missing coverage. Please review.

Project coverage is 78.64%. Comparing base (6e84834) to head (511b255).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-encoding/src/repdef.rs	91.49%	64 Missing and 9 partials ⚠️
rust/lance-encoding/src/format.rs	70.96%	9 Missing ⚠️
rust/lance-encoding/src/testing.rs	28.57%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3200      +/-   ##
==========================================
+ Coverage   78.62%   78.64%   +0.02%     
==========================================
  Files         243      244       +1     
  Lines       82889    83957    +1068     
  Branches    82889    83957    +1068     
==========================================
+ Hits        65170    66029     +859     
- Misses      14933    15137     +204     
- Partials     2786     2791       +5     

Flag	Coverage Δ
unittests	78.64% <91.07%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.