feat: add RTree index spec in table format by ddupg · Pull Request #5360 · lance-format/lance

ddupg · 2025-11-27T10:05:22Z

This PR proposes adding the R-Tree index specification to the Lance table format. For implementation details please see #5034

Feel free to leave comments or share feedback

chatgpt-codex-connector · 2025-11-27T10:05:26Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

github-actions · 2025-11-27T10:05:41Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

ddupg · 2025-12-03T03:04:12Z

@jackye1995 This PR is ready for review, PTAL when you have time.

codecov · 2025-12-05T10:12:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

jackye1995

added some comments

jackye1995 · 2025-12-08T07:43:31Z

+
+The R-Tree index is a static, immutable 2D spatial index. It is built on bounding boxes to organize the data. This index is intended to accelerate rectangle-based pruning.
+
+It is designed a multi-level hierarchical structure: leaf pages store tuples `(bbox, id=rowid)` for indexed geometries; branch pages aggregate child bounding boxes and store `id=pageid` pointing to child pages; a single root page encloses the entire dataset. Conceptually, it can be thought of as an extension of the B+-tree to multidimensional objects, where bounding boxes act as keys for spatial pruning.


nit: "It is designed as a multi-level hierarchical structure"

jackye1995 · 2025-12-08T07:44:44Z

+
+| Column | Type      | Nullable | Description                                                                                                                                                                                                                                                                     |
+|:-------|:----------|:---------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `bbox` | RectType  | false    | Type is Rect defined by [geoarrow-rs](https://github.com/geoarrow/geoarrow-rs) RectType; physical storage is Struct<xmin: {FloatType}, ymin: {FloatType}, xmax: {FloatType}, ymax: {FloatType}>. Represents the node bounding box (leaf: item bbox; branch: child aggregation). |


should make it clear if it is float32 or float64

jackye1995 · 2025-12-08T07:48:34Z

+
+| Column  | Type   | Nullable | Description                                                |
+|:--------|:-------|:---------|:-----------------------------------------------------------|
+| `nulls` | Binary | true     | Serialized RowIdTreeMap of rows with null/invalid geometry |


this is nullable, what does it mean to have a null value?

My mistake, this should be non-nullable

ddupg · 2025-12-09T08:24:39Z

Thank @jackye1995 for review. I've made the changes based on your comments.

jackye1995 · 2025-12-11T17:09:33Z

+
+Hilbert sorting imposes a linear order on 2D items using a space-filling Hilbert curve to maximize locality in both axes. This improves leaf clustering, which benefits query pruning.
+
+Items are mapped to a 16‑bit grid per axis (0..65535) by proportionally normalizing bbox centers within the global bounding box. Using a fast 2D Hilbert algorithm, we compute a 32‑bit `u32` key for each item and sort items in ascending order by this Hilbert value.


I think we should also describe the "fast 2D Hilbert algorithm" similar to how you describe the traversal. The goal of this document is that there is enough information such that someone can look at the doc and implement another version of this same index.

Thank @jackye1995 for the further explanation. I have added more details about Hilbert sorting along with pseudocode.

Change-Id: I60a33514d4d8e6914ae7df50eeb01e888ffb8e51

Change-Id: Iecdf94f858be5deb1e2ef0f9077f624b9fd755f5

Change-Id: I98174a8f5575681d97d1fbedbe57b234addcbe11

Change-Id: I84fba8e9e4f4569eb582ead663201ab0aaaf2bb6

jackye1995

Approving as the vote has passed

This PR proposes adding the R-Tree index specification to the Lance table format. For implementation details please see lance-format#5034 Feel free to leave comments or share feedback

ddupg mentioned this pull request Nov 27, 2025

Tracking issue for Geo type and index #5035

Open

3 tasks

ddupg changed the title ~~spec: add RTree index in table format~~ feat: add RTree index in table format Nov 27, 2025

github-actions Bot added the enhancement New feature or request label Nov 27, 2025

ddupg force-pushed the feat/rtree-spec branch from 03e1559 to 6bfba9e Compare November 27, 2025 10:14

ddupg changed the title ~~feat: add RTree index in table format~~ feat: add RTree index spec in table format Nov 27, 2025

ddupg mentioned this pull request Nov 27, 2025

feat: support GEO RTree index #5034

Merged

jackye1995 reviewed Dec 5, 2025

View reviewed changes

Comment thread docs/src/format/table/index/scalar/rtree.md

ddupg force-pushed the feat/rtree-spec branch from 0052764 to 5745d46 Compare December 5, 2025 09:24

jackye1995 reviewed Dec 8, 2025

View reviewed changes

ddupg force-pushed the feat/rtree-spec branch from 512026e to 43a2dac Compare December 9, 2025 06:25

ddupg requested a review from jackye1995 December 9, 2025 09:03

jackye1995 reviewed Dec 11, 2025

View reviewed changes

ddupg force-pushed the feat/rtree-spec branch from 97c103f to 6667bd7 Compare December 15, 2025 08:37

ddupg added 3 commits December 15, 2025 16:37

feat: add RTree index spec in table format

f4b6b1e

Change-Id: I60a33514d4d8e6914ae7df50eeb01e888ffb8e51

add RTreeIndexDetails

efad103

Change-Id: Iecdf94f858be5deb1e2ef0f9077f624b9fd755f5

address comments

f349a44

Change-Id: I98174a8f5575681d97d1fbedbe57b234addcbe11

ddupg force-pushed the feat/rtree-spec branch 3 times, most recently from c34606a to 0d23c8f Compare December 15, 2025 08:58

ddupg requested a review from jackye1995 December 15, 2025 08:59

more details about hilbert sorting

f93d063

Change-Id: I84fba8e9e4f4569eb582ead663201ab0aaaf2bb6

ddupg force-pushed the feat/rtree-spec branch from 0d23c8f to f93d063 Compare December 15, 2025 13:12

jackye1995 approved these changes Dec 20, 2025

View reviewed changes

jackye1995 merged commit d7504c2 into lance-format:main Dec 20, 2025
26 checks passed

andrea-reale mentioned this pull request Mar 30, 2026

emilk/fix write starvation rerun-io/lance#12

Closed


		The R-Tree index is a static, immutable 2D spatial index. It is built on bounding boxes to organize the data. This index is intended to accelerate rectangle-based pruning.

		It is designed a multi-level hierarchical structure: leaf pages store tuples `(bbox, id=rowid)` for indexed geometries; branch pages aggregate child bounding boxes and store `id=pageid` pointing to child pages; a single root page encloses the entire dataset. Conceptually, it can be thought of as an extension of the B+-tree to multidimensional objects, where bounding boxes act as keys for spatial pruning.


		Hilbert sorting imposes a linear order on 2D items using a space-filling Hilbert curve to maximize locality in both axes. This improves leaf clustering, which benefits query pruning.

		Items are mapped to a 16‑bit grid per axis (0..65535) by proportionally normalizing bbox centers within the global bounding box. Using a fast 2D Hilbert algorithm, we compute a 32‑bit `u32` key for each item and sort items in ascending order by this Hilbert value.

Conversation

ddupg commented Nov 27, 2025

Uh oh!

chatgpt-codex-connector Bot commented Nov 27, 2025

Uh oh!

github-actions Bot commented Nov 27, 2025

Uh oh!

ddupg commented Dec 3, 2025

Uh oh!

Uh oh!

codecov Bot commented Dec 5, 2025

Codecov Report

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

jackye1995 Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

jackye1995 Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jackye1995 Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

ddupg Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ddupg commented Dec 9, 2025

Uh oh!

jackye1995 Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

ddupg Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants