Skip to content

branch-3.0: [Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192 #47693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 25, 2025
Merged

branch-3.0: [Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192 #47693

dataroaring merged 1 commit into from
Feb 25, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Feb 10, 2025

Cherry-picked from #47192

@CalvinKirs
[Fix](catalog)Fixes query failures for Paimon tables stored in Kerber…
ec96975 
…ized HDFS (#47192)

### What problem does this PR solve?
Using JNI to directly read Paimon tables can lead to query failures when
the Paimon storage is on HDFS with Kerberos authentication enabled.

#### Reproduction Steps:
- Create a Paimon catalog stored on an HDFS cluster with Kerberos
authentication enabled.
- Execute the command: SET force_jni_scanner=true;.
- To ensure a clean environment, restart the BE (Backend) service.
- Perform any query on a table within the catalog.

```
2025-01-18 09:25:06  WARN Thread-13 org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:126) - Failed to open paimon_scanner: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
        at org.apache.doris.paimon.PaimonTableCache.getTable(PaimonTableCache.java:84)
        at org.apache.doris.paimon.PaimonJniScanner.initTable(PaimonJniScanner.java:237)
        at org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:122)
Caused by: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:708)
        at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:48)
        at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76)
        at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66)
        at org.apache.doris.paimon.PaimonTableCache.createCatalog(PaimonTableCache.java:75)
        at org.apache.doris.paimon.PaimonTableCache.loadTable(PaimonTableCache.java:58)
        at org.apache.doris.paimon.PaimonTableCache.access$000(PaimonTableCache.java:38)
        at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:51)
        at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:48)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
        ... 6 more
```
#### changes

This PR addresses an issue where queries fail when attempting to
directly read Paimon tables using JNI, specifically in environments
where HDFS is used as the storage backend and Kerberos authentication is
enabled. The failure is caused by the lack of proper Kerberos
authentication handling in the JNI implementation.
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 10, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 10, 2025

run buildall

dataroaring
dataroaring approved these changes Feb 25, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 704047c into branch-3.0 Feb 25, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-47192-branch-3.0 branch February 25, 2025 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @dataroaring @CalvinKirs