fix: open additional storage options provider related apis in lance dataset

[lance-community]

Backport of #5869

This PR backports the changes from the original PR to the release/v2.0 branch.

[github-actions]

PR Review

Summary

This backport of #5869 propagates the storage_options_provider attribute through various LanceDataset construction paths and adds a new_file_session() method for credential management.

P1 - Formatting Issue

Missing blank line between methods in the diff:

+            storage_options_provider=self._storage_options_provider,
+        )
     def checkout_version(

There should be a blank line between new_file_session() and checkout_version() to follow PEP 8 style guidelines. This may cause linting failures.

Test Coverage

The PR adds a new public API (new_file_session()) without corresponding tests. While the original PR (#5869) also lacked tests for this method, it would be valuable to add at least a basic test to verify the method returns a properly configured LanceFileSession.

---

Generated by Claude Code review