refactor(java): arrow schema should be returned by Fragment.mergeColumns

[fangbo]

1. The Fragment.mergeColumns's returned schema will be used in org.lance.operation.Merge which needs a Arrow Schema. So, Fragment.mergeColumn returns a Arrow Schema can avoid the convert from Lance Schema to Arrow Schema in Java. Like Dataset.getSchema, the convert is implemented in Rust.

2. Fix the issue Bug: Failed to add column with backfill for FixedSizeList lance-spark#138

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[fangbo]

@jackye1995 This PR is ready, could you please review it? Thank you.

[majin1102]

Thanks for catching this.

I wonder if we should fix the LanceSchema converting issue since the problem occurs there. Is there any blocking pointing that we could not use LanceSchema here?

[fangbo]

Thank you for comments.

I think LanceSchema should be fixed. It is another thing, we could submit another PR to fix it.

For Fragment.mergeColumns, it is sensible to return an Arrow schema, for three reasons:

1. The conversion from Lance schema to Arrow schema is already correctly implemented in Rust, so we can just reuse it like the Dataset.getSchema.
2. The Merge transaction itself expects an Arrow schema.
3. Dataset's public methods use Arrow for data operation (read/write) . So Arrow schema can keep consistency with other methods.

[majin1102]

Thanks for working on this. I wonder if we could fix LanceSchema convering issue there?

[fangbo]

This PR is duplicated with #5509 , close it.