google-cloud-nio: calling Files.exists(path) on a non-existing object in a requester pays bucket throws an exception

[skashin]

I used Java 1.8 on Linux, google-cloud-nio:0.87.0-alpha

To access a public requester pays bucket, I setup CloudStorageFileSystemProvider's default configuration with a google project name, and if I then call Files.exists(path) on an existing object path within this bucket, the call return true.

However, if I call Files.exists(path) on a path within that bucket that points to a non-existing object, this call throws an exception:
com.google.cloud.storage.StorageException: Bucket is requester pays bucket but no user project provided.
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:368)
at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:355)
at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:352)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:351)
at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:307)
at com.google.cloud.storage.contrib.nio.CloudStoragePath.seemsLikeADirectoryAndUsePseudoDirectories(CloudStoragePath.java:107)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.checkAccess(CloudStorageFileSystemProvider.java:703)
at java.nio.file.Files.exists(Files.java:2385)
at test.RequestorPaysBucketTest.runTests(RequestorPaysBucketTest.java:32)
at test.RequestorPaysBucketTest.main(RequestorPaysBucketTest.java:18)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Bucket is requester pays bucket but no user project provided.",
"reason" : "required"
} ],
"message" : "Bucket is requester pays bucket but no user project provided."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:401)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1089)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:358)

Below is the test Java class that reproduces this problem (EXISTING_OBJECT_PATH and NON_EXISTING_OBJECT_PATH need to be filled in and the google project name passed in through an environment variable GOOGLE_REQ_PAYS_PROJECT):

package test;

import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

import com.google.cloud.storage.contrib.nio.CloudStorageConfiguration;
import com.google.cloud.storage.contrib.nio.CloudStorageConfiguration.Builder;
import com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider;

public class RequestorPaysBucketTest {

private final String EXISTING_OBJECT_PATH = ...;
private final String NON_EXISTING_OBJECT_PATH = ...;

public static void main(String[] args) {
    (new RequestorPaysBucketTest()).runTest();
}

void runTest() {
    try {
        String requestorPaysProject = System.getenv().get("GOOGLE_REQ_PAYS_PROJECT");
        Builder builder = CloudStorageConfiguration.builder();
        builder = builder.autoDetectRequesterPays(true).userProject(requestorPaysProject);
        builder.usePseudoDirectories(true);
        CloudStorageFileSystemProvider.setDefaultCloudStorageConfiguration(builder.build());

        Path existingPath = Paths.get(new URI(EXISTING_OBJECT_PATH));
        System.out.println("Exists? " + Files.exists(existingPath));
        Path nonExistingPath = Paths.get(new URI(NON_EXISTING_OBJECT_PATH));
        System.out.println("Exists? " + Files.exists(nonExistingPath));
    } catch (Exception ex) {
        ex.printStackTrace();
    }
}

}

[JesseLovelace]

Thanks for bringing this to our attention. There's a case where a list call is made and the project isn't included. #4996 should fix this