Upgrade to Cascading version 2.6.3

[tdyas]

Upgrade to Cascading version 2.6.3.

[johnynek]

@tdyas what is this trying to do? Given that 2.6.3 is compatible (I hope) with 2.6.1, why force the upgrade here and not where downstream consumers make this choice?

At twitter, we often see some corner case bug with a cascading upgrade, so we are a bit conservative about bumping the version (we actually are still running 2.5 internally).

[tdyas]

I believe libraries make a statement (even unintentionally) when they pick a particular versions to use of a dependency. In this case, Scalding could be either making the statement that it works only with Cascading 2.6.1 or that Scalding is compatible with Cascading 2.6.x. (with 2.6.1 being the latest patch level at the time the SBT project was updated).

If it is the latter case, I believe a library should choose the latest patch level if there is no known issues.

My company has been running Scalding 0.13.1 / Cascading 2.6.3 in production since early March. Sufficient in my belief to bump the Scalding dependency on Cascading.

Plus I prefer to minimize overriding the versions of transitive dependencies.

The issue would be different for upgrading to Cascading 2.7.x or if I hadn't seen run in production for a month.

[tdyas]

And the Travis failure appears to be an intermittent jar download failure in Ivy.

[yingbo]

Why not upgrade to the newest library if all tests past.

If there are "some corner case bug with a cascading upgrade", they should be covered by tests, shouldn't they?

[johnynek]

Sadly, they are uncovered by tests at Twitter (with some issues around hash
joins). We actually don't even run 2.6 at all.

Users should be able to use a later version of cascading 2.X.Y in their own
build. There is an argument that we should depend on the lowest compatible
version with the scalding source and let users pick which later version of
cascading to use (of course that punts the decision to the user, but since
we don't have 100% coverage for realistic cluster environments, I'm not
sure we can do better right now).

I think the focus should be in upgrading to cascading 3 so we can use Tez
and Flink. There are some blockers on this see #1446.

On Tuesday, December 29, 2015, Yingbo Miao notifications@github.com wrote:

Why not upgrade to the newest library if all tests past.

If there are "some corner case bug with a cascading upgrade", they should
be covered by tests, shouldn't they?

—
Reply to this email directly or view it on GitHub
#1245 (comment).

P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

[sritchie]

Closing, since we have some PRs that are trying to get version 3 going.