Don't download from (unreliable) maven.org

[pdillinger]

Summary: I set up a mirror of our Java deps on github so we can download
them through github URLs rather than maven.org, which is proving
terribly unreliable from Travis builds.

Also sanitized calls to curl, so they are easier to read and
appropriately fail on download failure.

Test plan: CI

[facebook-github-bot]

@pdillinger has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[zhichao-cao]

Thanks!

[adamretter]

@pdillinger I am not too sure about this.

I was about to send a patch to replace the single remaining CENTRAL_REPO_URL, with the repo1 URL. As that seemed to be issue that I was seeing where downloading assertj sometimes failed.

Also, I am not sure of the merit of hosting binaries in git. Also I noticed that you used your own repo, this makes it very hard for us to update these dependencies in future. For example, there are newer versions of these libraries that we should likely move to shortly. I think it would be best to stay with Maven Central (after all it's used by almost every Java build on the planet), and try and resolve the stability issue.

[pdillinger]

The binaries are not in git. They are GitHub published artifacts, by creating a fake release on my fork. I can add a comment about that.

All of the maven.org downloads have been failing very often (from Travis) not just assertj.

If you need to update in the future, that would be to change version and since version is in the URL, you have to change the URL anyway. The next person to update can find some reliable place to put them (preferably as GitHub artifacts, see below), and update the URL.

We have no structural reason here to use maven.org (e.g. a maven build), so I see sticking with it as a minor convenience on updating packages, weighed against majorly disruptive and recurring unreliability.

I can't easily confirm but I suspect maven.org might be throttling traffic from Travis because it's such a heavy user. But we had trouble with reliability of maven fetches at my last job too, without Travis.

We already use GitHub urls to download non-java packages (not to mention checkout), so it's the natural place to fetch from if possible to maximize build reliability.

[pdillinger]

From adamretter (previously accidentally edited into my comment):

Okay but the "my fork" is the bit that worries me. No one can update this apart from you, in addition you may at some point leave the Rocks team (hopefully not).

I addressed that:

(me) If you need to update in the future, that would be to change version and since version is in the URL, you have to change the URL anyway. The next person to update can find some reliable place to put them (preferably as GitHub artifacts, see below), and update the URL.

(adamretter) I think we need a better place for these if we are moving off of Maven Central.

I agree, but I don't think I have the privileges to put them on Facebook's account unless they're in revision control (ew). This commit is intended as a solution until someone implements a better one.

[facebook-github-bot]

@pdillinger has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[pdillinger]

Example non-assertj failures (just a small sample of the history):

https://travis-ci.org/facebook/rocksdb/jobs/643569274
https://travis-ci.org/facebook/rocksdb/jobs/643562137
https://travis-ci.org/facebook/rocksdb/jobs/643562142
https://travis-ci.org/facebook/rocksdb/jobs/643527616

The maven.org unreliability might be healthy again, so I won't push until seeing about a better hosting place or we see more failures.

[pdillinger]

Looks like today just assertj is failing. I'm going to push this and work with Fosco on a better solution.

[pdillinger]

Looks like today just assertj is failing. I'm going to push this and work with Fosco on a better solution.

And probably then backport to previous branches

[facebook-github-bot]

@pdillinger merged this pull request in 90c71aa.