[C++] Make head optional in s3fs

[asfimport]

When you open an input file with the f3fs, it issues a head request to S3 to check if the file is present/authorized and get the size (

arrow/cpp/src/arrow/filesystem/s3fs.cc

Line 407 in f16f76a

auto outcome = client_->HeadObject(req);

).

This call comes with a non-neglictable cost:

• adds latency

• priced the same as a GET request by AWS

  I fail to see usecases where this call is really crucial:

• if the file is not present/authorized, failing at first read seems to have mostly the same effect as failing on opening. I agree that it is kind of "usual" for an open call to fail eagerly, so to avoid surprises we could add a flag indicating if we don't need to fail when running OpenInputFile on an inaccessible file.

• getting the size can be done on the first read, and could be mostly avoided on caller side if the filesystem api provided read-from-end capabilities (compatible with fs reads using ios::end and on http filesystems with bytes=-xxx). Worst case scenario the call to head could be done lazily when calling getSize().

  I agree that it makes things a bit more complex, and I understand that you would not want to complexify the generic fs api because of blob storage behavior. But obviously there are workloads where this has a significant impact.

Reporter: Rémi Dettai / @rdettai
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #7547

_{Note: This issue was originally created as ARROW-8950. Please see the migration documentation for further details.}

[asfimport]

Rémi Dettai / @rdettai:
@pitrou Do you have any idea on how to address this (or if it should be addressed at all ;)) ?

[asfimport]

Antoine Pitrou / @pitrou:
Would it be ok to be able to disable it in S3Options?

[asfimport]

Rémi Dettai / @rdettai:
You need more than this. If you don't have a "read from end" type function in the filesystem API, you will still need to get the size first in order to read the end of the file. The primary usecase for this is of course parquet, where you need to read the footer first.

A workaround if we don't want to extend the generic filesystem API would be to provide the file size manually when opening the file, this way you could use file sizes you got in batches with list commands or from some kind of catalog.

[asfimport]

Antoine Pitrou / @pitrou:
Thanks for the explanation. Then I think we may add an OpenInputFile overload that takes a FileInfo argument, so that the user can pass the file size (it will be trusted to be correct).

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 7547
#7547