[C++][CI] arrow-s3fs-test fails on some nightly jobs #36447
Description
Describe the bug, including details regarding any error messages, version, and platform.
It seems that the arrow-s3fs-test fails with file exists for some of the tests:
Error: mkdir /tmp/s3fs-test-9kk8hwry/bucket/somefile/: file exists (*fs.PathError)
This is happening on:
- test-alpine-linux-cpp
- test-debian-11-cpp-amd64
- test-skyhook-integration
- test-ubuntu-20.04-cpp
- test-ubuntu-22.04-cpp
The full error log:
75/108 Test #69: arrow-s3fs-test ...........................***Failed 21.30 sec
Running arrow-s3fs-test, redirecting output into /build/cpp/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
/arrow/cpp/build-support/run-test.sh: line 88: 30269 Aborted (core dumped) $TEST_EXECUTABLE "$@" > $LOGFILE.raw 2>&1
Running main() from /home/buildozer/aports/main/gtest/src/googletest-14aa11db02d9851d957f93ef9fddb110c1aafdc6/googletest/src/gtest_main.cc
[==========] Running 62 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from S3OptionsTest
[ RUN ] S3OptionsTest.FromUri
[ OK ] S3OptionsTest.FromUri (579 ms)
[ RUN ] S3OptionsTest.FromAccessKey
[ OK ] S3OptionsTest.FromAccessKey (132 ms)
[ RUN ] S3OptionsTest.FromAssumeRole
[ OK ] S3OptionsTest.FromAssumeRole (135 ms)
[----------] 3 tests from S3OptionsTest (847 ms total)
[----------] 4 tests from S3RegionResolutionTest
[ RUN ] S3RegionResolutionTest.PublicBucket
[ OK ] S3RegionResolutionTest.PublicBucket (372 ms)
[ RUN ] S3RegionResolutionTest.RestrictedBucket
[ OK ] S3RegionResolutionTest.RestrictedBucket (341 ms)
[ RUN ] S3RegionResolutionTest.NonExistentBucket
[ OK ] S3RegionResolutionTest.NonExistentBucket (1490 ms)
[ RUN ] S3RegionResolutionTest.InvalidBucketName
[ OK ] S3RegionResolutionTest.InvalidBucketName (135 ms)
[----------] 4 tests from S3RegionResolutionTest (2340 ms total)
[----------] 2 tests from S3FileSystemRegionTest
[ RUN ] S3FileSystemRegionTest.Default
[ OK ] S3FileSystemRegionTest.Default (135 ms)
[ RUN ] S3FileSystemRegionTest.EnvironmentVariable
[ OK ] S3FileSystemRegionTest.EnvironmentVariable (136 ms)
[----------] 2 tests from S3FileSystemRegionTest (271 ms total)
[----------] 1 test from TestMinioServer
[ RUN ] TestMinioServer.Connect
[ OK ] TestMinioServer.Connect (706 ms)
[----------] 1 test from TestMinioServer (706 ms total)
[----------] 26 tests from TestS3FS
[ RUN ] TestS3FS.GetFileInfoRoot
[ OK ] TestS3FS.GetFileInfoRoot (197 ms)
[ RUN ] TestS3FS.GetFileInfoBucket
[ OK ] TestS3FS.GetFileInfoBucket (244 ms)
[ RUN ] TestS3FS.GetFileInfoObject
[ OK ] TestS3FS.GetFileInfoObject (300 ms)
[ RUN ] TestS3FS.GetFileInfoSelector
[ OK ] TestS3FS.GetFileInfoSelector (257 ms)
[ RUN ] TestS3FS.GetFileInfoSelectorRecursive
[ OK ] TestS3FS.GetFileInfoSelectorRecursive (277 ms)
[ RUN ] TestS3FS.GetFileInfoGenerator
[ OK ] TestS3FS.GetFileInfoGenerator (240 ms)
[ RUN ] TestS3FS.CreateDir
API: PutObject(bucket=bucket, object=somefile/)
Time: 01:32:48 UTC 07/04/2023
DeploymentID: 010901b4-1aed-45ad-a0ee-e0af17423c04
RequestID: 176E861CC5143A4A
RemoteHost: 127.0.0.1
Host: 127.0.0.8:54373
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: mkdir /tmp/s3fs-test-9kk8hwry/bucket/somefile/: file exists (*fs.PathError)
5: internal/logger/logger.go:278:logger.LogIf()
4: cmd/fs-v1.go:1036:cmd.(*FSObjects).putObject()
3: cmd/fs-v1.go:1012:cmd.(*FSObjects).PutObject()
2: cmd/object-handlers.go:1830:cmd.objectAPIHandlers.PutObjectHandler()
1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
API: PutObject(bucket=bucket, object=somefile/)
Time: 01:32:48 UTC 07/04/2023
DeploymentID: 010901b4-1aed-45ad-a0ee-e0af17423c04
RequestID: 176E861CC5143A4A
RemoteHost: 127.0.0.1
Host: 127.0.0.8:54373
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: mkdir /tmp/s3fs-test-9kk8hwry/bucket/somefile/: file exists (*fs.PathError)
5: internal/logger/logger.go:278:logger.LogIf()
4: cmd/api-errors.go:2123:cmd.toAPIErrorCode()
3: cmd/api-errors.go:2148:cmd.toAPIError()
2: cmd/object-handlers.go:1832:cmd.objectAPIHandlers.PutObjectHandler()
1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
[ OK ] TestS3FS.CreateDir (335 ms)
[ RUN ] TestS3FS.DeleteFile
[ OK ] TestS3FS.DeleteFile (243 ms)
[ RUN ] TestS3FS.DeleteDir
[ OK ] TestS3FS.DeleteDir (294 ms)
[ RUN ] TestS3FS.DeleteDirContents
[ OK ] TestS3FS.DeleteDirContents (278 ms)
[ RUN ] TestS3FS.DeleteDirContentsAsync
API: PutObject(bucket=bucket, object=somefile/)
Time: 01:32:49 UTC 07/04/2023
DeploymentID: a9a8af82-9903-4a4b-9d24-8647ccf47604
RequestID: 176E861D0468B79A
API: CopyObject(bucket=s3fs-test-bucket, object=xxx/mno)
Time: 01:32:58 UTC 07/04/2023
DeploymentID: b07a70c3-9139-46de-abf3-6bdfc538bf1c
RequestID: 176E861F1AAC1E91
RemoteHost: 127.0.0.1
Host: 127.0.0.36:33197
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: file access denied (cmd.StorageErr)
6: internal/logger/logger.go:278:logger.LogIf()
5: cmd/fs-v1-helpers.go:360:cmd.fsRenameFile()
4: cmd/fs-v1.go:1107:cmd.(*FSObjects).putObject()
3: cmd/fs-v1.go:685:cmd.(*FSObjects).CopyObject()
2: cmd/object-handlers.go:1501:cmd.objectAPIHandlers.CopyObjectHandler()
1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
[ OK ] TestS3FSGeneric.MoveFile (380 ms)
[ RUN ] TestS3FSGeneric.MoveDir
/arrow/cpp/src/arrow/filesystem/test_util.cc:453: Skipped
Filesystem doesn't allow moving directories
[ SKIPPED ] TestS3FSGeneric.MoveDir (160 ms)
[ RUN ] TestS3FSGeneric.CopyFile
API: CopyObject(bucket=s3fs-test-bucket, object=def/mno)
Time: 01:32:58 UTC 07/04/2023
DeploymentID: d53f2a07-9c5c-4a8e-aa2a-e6ba8713c057
RequestID: 176E861F397DC956
RemoteHost: 127.0.0.1
Host: 127.0.0.38:52109
UserAgent: aws-sdk-cpp/1.10.55 Linux/5.15.0-1040-azure x86_64 GCC/11.2.1
Error: file access denied (cmd.StorageErr)
6: internal/logger/logger.go:278:logger.LogIf()
5: cmd/fs-v1-helpers.go:360:cmd.fsRenameFile()
4: cmd/fs-v1.go:1107:cmd.(*FSObjects).putObject()
3: cmd/fs-v1.go:685:cmd.(*FSObjects).CopyObject()
2: cmd/object-handlers.go:1501:cmd.objectAPIHandlers.CopyObjectHandler()
1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()
[ OK ] TestS3FSGeneric.CopyFile (368 ms)
[ RUN ] TestS3FSGeneric.GetFileInfo
[ OK ] TestS3FSGeneric.GetFileInfo (242 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoVector
[ OK ] TestS3FSGeneric.GetFileInfoVector (295 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoSelector
[ OK ] TestS3FSGeneric.GetFileInfoSelector (318 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
[ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (398 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoAsync
[ OK ] TestS3FSGeneric.GetFileInfoAsync (231 ms)
[ RUN ] TestS3FSGeneric.GetFileInfoGenerator
[ OK ] TestS3FSGeneric.GetFileInfoGenerator (298 ms)
[ RUN ] TestS3FSGeneric.OpenOutputStream
[ OK ] TestS3FSGeneric.OpenOutputStream (385 ms)
[ RUN ] TestS3FSGeneric.OpenAppendStream
/arrow/cpp/src/arrow/filesystem/test_util.cc:932: Skipped
Filesystem doesn't allow file appends
[ SKIPPED ] TestS3FSGeneric.OpenAppendStream (215 ms)
[ RUN ] TestS3FSGeneric.OpenInputStream
[ OK ] TestS3FSGeneric.OpenInputStream (288 ms)
[ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
[ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (280 ms)
[ RUN ] TestS3FSGeneric.OpenInputStreamAsync
[ OK ] TestS3FSGeneric.OpenInputStreamAsync (253 ms)
[ RUN ] TestS3FSGeneric.OpenInputFile
[ OK ] TestS3FSGeneric.OpenInputFile (265 ms)
[ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
[ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (281 ms)
[ RUN ] TestS3FSGeneric.OpenInputFileAsync
[ OK ] TestS3FSGeneric.OpenInputFileAsync (260 ms)
[ RUN ] TestS3FSGeneric.SpecialChars
[ OK ] TestS3FSGeneric.SpecialChars (301 ms)
[----------] 26 tests from TestS3FSGeneric (8043 ms total)
[----------] Global test environment tear-down
[==========] 62 tests from 6 test suites ran. (21022 ms total)
[ PASSED ] 60 tests.
[ SKIPPED ] 2 tests, listed below:
[ SKIPPED ] TestS3FSGeneric.MoveDir
[ SKIPPED ] TestS3FSGeneric.OpenAppendStream
pure virtual method called
terminate called without an active exception
/build/cpp/src/arrow/filesystem
Component(s)
C++, Continuous Integration