ARROW-6720: [C++] Add HDFS implementation to filesystem layer

[pitrou]

Based on initial work by @xuechendi.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6720

[pitrou]

@nealrichardson Will need your help for the R AppVeyor failure:
https://ci.appveyor.com/project/pitrou/arrow/builds/28838764#L3459

[nealrichardson]

Looks like ARROW_FILESYSTEM now requires a new dependency? Is it vendored?

[pitrou]

It's built by the Arrow build chain, but it's not in the packages used by R-Arrow ("rarow"?) on Windows. It seems to work elsewhere, including R Ubuntu here: https://github.com/apache/arrow/pull/5820/checks?check_run_id=301764483

[nealrichardson]

Yes but Windows is "special". I'll dig in a bit and see if I can understand what the deal is. Alternatively, is uriparser needed for all of filesystem or just HDFS?

[pitrou]

It's needed for all of filesystem (it's needed for Flight too).

[nealrichardson]

🤔 there's also this message higher up: https://ci.appveyor.com/project/pitrou/arrow/builds/28838764#L1576

[pitrou]

It says "Building uriparser from source" which is expected at this point.

[nealrichardson]

Ok, and on master it's not building uriparser (cf. https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/28838835/job/788gr6gydfhrdk4v?fullLog=true)

Have you tried without adding -luriparser here? https://github.com/apache/arrow/pull/5820/files#diff-eb7edbff9a6d44777b97bc545daa680dR44

Maybe it's bundled somehow and not needed? Otherwise, where does it get built to?

[nealrichardson]

@ursabot crossbow submit macos-r-autobrew

[pitrou]

Have you tried without adding -luriparser here?

Yes, it fails with a link error: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/28836148/job/o95eh8ft96n3n0pl#L3348

[ursabot]

AMD64 Conda Crossbow Submit (#76833) builder has been succeeded.

Revision: 9793132e05e5b674c492fe9e64f442239cd2266c

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-324

Task	Status
macos-r-autobrew

[nealrichardson]

Ok so we need it. We're building it and that seems not to error, but it's not getting bundled with the build. That is, we're dealing with https://github.com/apache/arrow/blob/master/ci/appveyor-build-r.sh, which calls https://github.com/apache/arrow/blob/master/ci/PKGBUILD and https://github.com/apache/arrow/blob/master/ci/windows-pkg-arrow-for-r.sh.

[pitrou]

Does it bundle other dependencies such as double-conversion?

[nealrichardson]

double-conversion is an external dependency, so that gets downloaded and bundled here: https://github.com/apache/arrow/blob/master/ci/windows-pkg-arrow-for-r.sh#L61

[nealrichardson]

If we don't want to use the vendored uriparser, we could port https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-uriparser/PKGBUILD to the rtools-packages repository and then treat it like we do double-conversion et al.

[pitrou]

If we don't want to use the vendored uriparser

I'm not sure what you mean by that. Personally, I have no preference :-) I just don't know how to do it.

[nealrichardson]

Neither do I :)

cf. #5814

[nealrichardson]

Locally it gets built into $BUILD_DIR/uriparser_ep-install/lib/liburiparser.a. Maybe we can grab that before makepkg-mingw deletes it.

[nealrichardson]

Looks like we have the same problem on macOS: https://travis-ci.org/ursa-labs/crossbow/builds/611598498#L831

[pitrou]

Can I get you to handle the issue?

[nealrichardson]

Realistically I can’t commit to doing it before going on vacation. But it doesn’t require me, there’s no other special R knowledge needed, I think we just need to mv the built liburiparser to the right directory.

[fsaintjacques]

This function should take the path as parameter and return FileStats directly, would make thing more natural.

[pitrou]

No, because it acts on an existing FileStats. I mean other members have been set by the caller.

[nealrichardson]

@pitrou I did some exploration in the Appveyor build and I think I see how to grab the liburiparser that we build; however, it seems that we build a .dll but for R we need a static library.

I see there is a uriparser cmake flag BUILD_SHARED_LIBS, which if it is OFF will build static libraries. Is there a way we can pass through that flag (and presumably have it just follow what's in the arrow cmake wrt shared/static)?

[bkietz]

@nealrichardson that flag is already set to off when building uriparser; it really looks like we're trying to always build it static

[nealrichardson]

And yet I see liburiparser-1.dll is what is made on Appveyor: https://ci.appveyor.com/project/nealrichardson/arrow/builds/28875888?fullLog=true#L1177

Locally I do see liburiparser.a though.

[nealrichardson]

Maybe there's another flag needed somewhere for mingw? https://stackoverflow.com/a/42979732/11897522

[bkietz]

there was a merge conflict due to ARROW-6633. I've rebased

[pitrou]

I've rebased for the uriparser vendoring, this should fix the R build on Windows.

[bkietz]

CI is green, I'll merge this now