ci(hf): isolate CI repos and stabilize binding tests

[Xuanwo]

Which issue does this PR close?

Closes #.
Related: #7367

Rationale for this change

HF behavior tests are currently unstable in CI for two different reasons. The Java binding still crashes on Linux in the HF blocking path, and the shared HF CI repositories introduce cross-job interference for writable test cases.

What changes are included in this PR?

This PR switches HF CI jobs to create and clean up temporary repos or buckets per job so writable behavior tests do not race on the same remote state. It also keeps the Java HF behavior case disabled while #7367 is still open, and restores the missing create_dir capability gating in the Go and Node.js HF bucket list suites.

Are there any user-facing changes?

No.

AI Usage Statement

Used GPT-5 for CI log inspection, patch preparation, and PR description drafting.

[kszucs]

Seems like within single jobs there are still commits races. @Xuanwo could you try making condition not match retryable?

diff --git a/core/services/hf/src/error.rs b/core/services/hf/src/error.rs
index c0e36616d..84c83f8e5 100644
--- a/core/services/hf/src/error.rs
+++ b/core/services/hf/src/error.rs
@@ -44,7 +44,7 @@ pub(super) fn parse_error(resp: Response<Buffer>) -> Error {
     let (kind, retryable) = match parts.status {
         StatusCode::NOT_FOUND => (ErrorKind::NotFound, false),
         StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false),
-        StatusCode::PRECONDITION_FAILED => (ErrorKind::ConditionNotMatch, false),
+        StatusCode::PRECONDITION_FAILED => (ErrorKind::ConditionNotMatch, true),
         StatusCode::INTERNAL_SERVER_ERROR
         | StatusCode::BAD_GATEWAY
         | StatusCode::SERVICE_UNAVAILABLE

[Xuanwo]

Seems like within single jobs there are still commits races. @Xuanwo could you try making condition not match retryable?

Thanks! And updated.

[kszucs]

@Xuanwo this retry mechanism also tries to upload the xet blobs again raising an Already completed error. It would be better to enable retries just for the committing phase, though that would require custom retry logic with backon.

[kszucs]

Could you try to cherry pick fa165d5 ?

[Xuanwo]

@Xuanwo this retry mechanism also tries to upload the xet blobs again raising an Already completed error. It would be better to enable retries just for the committing phase, though that would require custom retry logic with backon.

I believe the root cause here is our writer's close is not re-enter safe. So I think the better solution is to store the finished file info and retry the commit.

[kszucs]

I can try to make it re-enter safe e.g. using a state enum. Let my try that.

[Xuanwo]

I can try to make it re-enter safe e.g. using a state enum. Let my try that.

Thank you for that! But I will cover this 🤟

[Xuanwo]

Only one test failure now, working on it

failures:

---- behavior::test_remove_all_basic ----
test panicked: all objects should be removed


failures:
    behavior::test_remove_all_basic

test result: FAILED. 91 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 27.53s

[Xuanwo]

Cool, let's go!