fix: make column name lookups case-insensitive by wjones127 · Pull Request #5465 · lance-format/lance

wjones127 · 2025-12-12T21:27:59Z

Summary

SQL parsers lowercase unquoted identifiers, but Arrow schema lookups are case-sensitive
This caused mixed-case column names (e.g., userId) to fail in filter expressions, scalar index creation, and merge insert
Added case-insensitive field lookup that tries exact match first, then falls back to case-insensitive match

Test plan

Added Python tests for mixed-case column names (python/tests/test_column_names.py)
All 14 new tests pass
Existing Rust tests pass (68 merge_insert tests, 45 lance-datafusion tests)

🤖 Generated with Claude Code

SQL parsers lowercase unquoted identifiers, but Arrow schema lookups are case-sensitive. This caused mixed-case column names (e.g., `userId`) to fail in filter expressions, scalar index creation, and merge insert. Added case-insensitive field lookup that tries exact match first, then falls back to case-insensitive match. Applied this to: - SQL filter/expression parsing in Planner - Scalar index column validation - Merge insert key column resolution and join expressions Fixes lance-format#3424 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Move case-insensitive column name resolution directly into Planner::column() instead of having a separate post-processing step. This is cleaner since Planner already has access to the schema. Removed the resolve_column_names function from logical_expr.rs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The case-insensitive column resolution was incorrectly replacing nested field paths (like `meta.lang`) with just the leaf field name (`lang`). Now nested paths are kept intact while simple column names still get case-insensitive resolution. Also adds tests for mixed-case and special character names in nested fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

codecov · 2025-12-12T22:43:36Z

Codecov Report

❌ Patch coverage is 82.85714% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-core/src/datatypes/schema.rs	52.50%	17 Missing and 2 partials ⚠️
rust/lance/src/dataset/write/merge_insert.rs	91.86%	5 Missing and 2 partials ⚠️
rust/lance-core/src/datatypes/field.rs	95.45%	1 Missing ⚠️
rust/lance-datafusion/src/planner.rs	94.11%	1 Missing ⚠️
rust/lance/src/dataset/scanner.rs	85.71%	1 Missing ⚠️
...ce/src/dataset/write/merge_insert/assign_action.rs	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Added explain_plan() checks to all scalar index tests to verify the index is actually being used in the query plan by checking for "ScalarIndexQuery". 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chatgpt-codex-connector · 2025-12-13T00:11:14Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

westonpace

I'm unsure if this gives us consistent "try exact first and the case insensitive" behavior everywhere. Also, it might be nice to have test cases for multiple columns with different cases.

I think case-insensitive but inconsistent if multiple columns with same name is better than what we have today but having consistent behavior everywhere would be idea.

westonpace · 2025-12-18T18:51:32Z

+                "user-id": range(100),
+                "order:id": range(100, 200),
+                "item_name": [f"item_{i}" for i in range(100)],


This one is scary 😆 I wonder if we should limit the available character set for column names at all?

I actually did have a user who wanted this. I don't think there's any reason we shouldn't support it.

- Add Field::resolve_case_insensitive for nested field path resolution - Expose field_case_insensitive to Python bindings - Update Python SDK to use case-insensitive field lookup in index APIs - Fix format_field_path to quote fields with special characters - Add Rust test for merge_insert with mixed-case key column - Add Python test for lowercased nested path index creation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use non-correlated column values so tests fail if wrong column is resolved: - camelCase: 0-99 ascending - CamelCase: 99-0 descending - CAMELCASE: 50-99,0-49 rotated Each test now verifies values from multiple columns to ensure correct resolution. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- SQL parsers lowercase unquoted identifiers, but Arrow schema lookups are case-sensitive - This caused mixed-case column names (e.g., `userId`) to fail in filter expressions, scalar index creation, and merge insert - Added case-insensitive field lookup that tries exact match first, then falls back to case-insensitive match - [x] Added Python tests for mixed-case column names (`python/tests/test_column_names.py`) - [x] All 14 new tests pass - [x] Existing Rust tests pass (68 merge_insert tests, 45 lance-datafusion tests) Fixes lance-format#3424 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

- SQL parsers lowercase unquoted identifiers, but Arrow schema lookups are case-sensitive - This caused mixed-case column names (e.g., `userId`) to fail in filter expressions, scalar index creation, and merge insert - Added case-insensitive field lookup that tries exact match first, then falls back to case-insensitive match - [x] Added Python tests for mixed-case column names (`python/tests/test_column_names.py`) - [x] All 14 new tests pass - [x] Existing Rust tests pass (68 merge_insert tests, 45 lance-datafusion tests) Fixes #3424 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

## Summary - SQL parsers lowercase unquoted identifiers, but Arrow schema lookups are case-sensitive - This caused mixed-case column names (e.g., `userId`) to fail in filter expressions, scalar index creation, and merge insert - Added case-insensitive field lookup that tries exact match first, then falls back to case-insensitive match ## Test plan - [x] Added Python tests for mixed-case column names (`python/tests/test_column_names.py`) - [x] All 14 new tests pass - [x] Existing Rust tests pass (68 merge_insert tests, 45 lance-datafusion tests) Fixes lance-format#3424 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

wjones127 and others added 2 commits December 12, 2025 12:52

github-actions Bot added bug Something isn't working python labels Dec 12, 2025

wjones127 marked this pull request as ready for review December 13, 2025 00:11

westonpace approved these changes Dec 18, 2025

View reviewed changes

wjones127 and others added 3 commits December 18, 2025 11:38

fix nested fields

8369d3a

wjones127 merged commit 8f880a5 into lance-format:main Dec 18, 2025
28 of 30 checks passed

wjones127 deleted the fix-case-insensitive-column-names branch December 18, 2025 22:54

andrea-reale mentioned this pull request Mar 30, 2026

emilk/fix write starvation rerun-io/lance#12

Closed

wjones127 mentioned this pull request Apr 3, 2026

fix: quote column names in merge_insert join to preserve case sensiti… #6397

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: make column name lookups case-insensitive#5465

fix: make column name lookups case-insensitive#5465
wjones127 merged 7 commits intolance-format:mainfrom
wjones127:fix-case-insensitive-column-names

wjones127 commented Dec 12, 2025

Uh oh!

codecov Bot commented Dec 12, 2025 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Dec 13, 2025

Uh oh!

westonpace left a comment

Uh oh!

westonpace Dec 18, 2025

Uh oh!

wjones127 Dec 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wjones127 commented Dec 12, 2025

Summary

Test plan

Uh oh!

codecov Bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chatgpt-codex-connector Bot commented Dec 13, 2025

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

westonpace Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

wjones127 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Dec 12, 2025 •

edited

Loading