Skip to content

fix: pass dataset_options to SafeLanceDataset in worker processes#6278

Merged
eddyxu merged 3 commits intomainfrom
lei/torch_dataset_options
Mar 24, 2026
Merged

fix: pass dataset_options to SafeLanceDataset in worker processes#6278
eddyxu merged 3 commits intomainfrom
lei/torch_dataset_options

Conversation

@eddyxu
Copy link
Copy Markdown
Member

@eddyxu eddyxu commented Mar 24, 2026

Worker processes were opening the dataset without dataset_options, silently dropping any options (storage_options, version, index_cache_size, etc.) set by the caller. Also remove the debug print statement.

Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com

Worker processes were opening the dataset without dataset_options,
silently dropping any options (storage_options, version, index_cache_size,
etc.) set by the caller. Also remove the debug print statement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot added bug Something isn't working python labels Mar 24, 2026
@github-actions
Copy link
Copy Markdown
Contributor

LGTM. Clear bugfix — worker processes were silently dropping dataset_options and the debug print needed removal. The fix correctly mirrors what _safe_preload already does.

One minor note: there are no tests covering SafeLanceDataset worker initialization with custom dataset_options. A test that round-trips with e.g. a pinned version would guard against regression. Not blocking given the simplicity of the change.

@eddyxu eddyxu merged commit f2d2068 into main Mar 24, 2026
13 checks passed
@eddyxu eddyxu deleted the lei/torch_dataset_options branch March 24, 2026 19:19
westonpace pushed a commit that referenced this pull request Mar 24, 2026
)

Worker processes were opening the dataset without dataset_options,
silently dropping any options (storage_options, version,
index_cache_size, etc.) set by the caller. Also remove the debug print
statement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
wjones127 pushed a commit to wjones127/lance that referenced this pull request Mar 29, 2026
…nce-format#6278)

Worker processes were opening the dataset without dataset_options,
silently dropping any options (storage_options, version,
index_cache_size, etc.) set by the caller. Also remove the debug print
statement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants