Test & Fix for deadlock in Cascading-Local #47
Test & Fix for deadlock in Cascading-Local #47cchepelov wants to merge 60 commits intocwensel:wip-3.0from
Conversation
…rks to fully leverage declared field type information to reduce serialized data and perform bitwise equality comparisons. See c.t.h.TupleSerializationProps to disable bitwise comparisons.
…rms through the c.f.FlowRuntimeProps "cascading.flow.runtime.splits.combine" property. If enabled, will induce c.t.h.Hfs to enable combined files support on the MapReduce platforms.
…n, and a method #applyFields() to update field names.
…nd serialization dependencies and updated cascading-hadoop2-mr1 and cascading-hadoop2-tez to depend on cascading-hadoop2-io.
…operations based on Janino and it isolate the Janino dependency. A dependency to cascading-expression must be added to projects that depend on the isolated classes.
…nsert After and not AfterEachEdge.
…ors to store additional metadata on c.f.FlowStep instances.
…splicing back into a c.p.HashJoin could create an invalid plan.
…r methods on element graphs
…imeline server dependencies.
…lag to disable services jar
…g, relax CascadingStats for sub-classing
|
Back on
JoinFieldedPipesPlatformTest.testForkThenJoin returns zero items on tez, whereas it returns 5 items on -local, -hadoop, -hadoop-mr1. @cwensel , I'm afraid (unless I failed something) I've found a tez vs. rest of the world discrepancy. JoinFieldedPipesPlatformTest.testForkCoGroupThenHashJoinCoGroupAgain fails clusterside, because of a missed copyFromLocal() of one input file (strange that it now passes). ==> pushing a quick commit for that. (update: now down to JoinFieldedPipesPlatformTest.testForkThenJoin which fails only under tez. Identical behaviour between java7u91 and java8u72) |
cchepelov
force-pushed
the
local_fork_hashjoin_deadlock
branch
from
0c7ec68 to
ba5fde3
Compare
cchepelov
force-pushed
the
local_fork_hashjoin_deadlock
branch
from
55463d6 to
43bb327
Compare
|
Patch updated and smashed back into 1 commit. |
|
JoinFieldedPipesPlatformTest.testForkCoGroupThenHashJoinCoGroupAgain failed again on Tez (linux and java 7). Are you sure, you send me the latest code? |
|
What? Double checking: Will advise once it's complete (cc @simondumas) Update: toolchain manifest: |
|
Same on the Oracle 7u80 jdk: (snip) I guess at this stage, I'm stuck… |
…m rules possibly preventing some 'self joins'
|
I’ve fixed the known underlying issues in wip-3.1. as for this patch, is there anyway we can not add new test functions and use existing ones like Identity or ExpressionFunction. would be nice to keep this class from growing too large. with those changes we an re-test and make the commit/push/publish. |
cchepelov
force-pushed
the
local_fork_hashjoin_deadlock
branch
from
43bb327 to
9624012
Compare
|
|
I think some cleanup needs to happen here. the patch doesn’t apply, might have to do with the fact its 1.6M and attempts to update every file in my wip-3.1 branch. update: let me just apply the singular commit.. |
|
Le 15/01/2016 19:21, Chris K Wensel a écrit :
|
cchepelov
force-pushed
the
local_fork_hashjoin_deadlock
branch
from
9624012 to
ea14fb2
Compare
…g. Scalding's SkewedJoin)
(fix expected output)
cascading-local: use ParallelFork to avoid deadlocks in Fork-HashJoin scenarios
java7 compatibility
(to be fair, workStealingThreadPool worked by chance, as there are more cores on my machine than the maximum amount of inbound pipes in the joins present downstream from a fork in the test suite)
(forgotten test input file needs to go to the platform)
Disabling the cascading.JoinFieldedPipesPlatformTest.testForkThenJoin test, which fails under Tez (for an unrelated cause)
remove spurious janino import
removed custom functions, replaced by ExpressionFunctions
Author: Cyrille Chépélov (TP12) <cch@transparencyrights.com>
Date: Thu Jan 7 10:44:04 2016 +0100
cchepelov
force-pushed
the
local_fork_hashjoin_deadlock
branch
from
ea14fb2 to
e2ba161
Compare
|
Turns out I started this on the wrong branch (wip-3.0). While I could rebase my branch off wip-3.1, I can't rebase the PR itself, I have to open a new one. |
|
The action continues here: #48 |
6 participants
When running scalding's test suite, a test freezes and dies (SkewJoinPipeTest)
This PR provides a simplified test case that exhibits the same blocking behaviour + provides a fix
The fix works by splitting all edges downstream from a Fork into a separate thread, on the (fair) assumption that downstream consumers are already thread-hardened anyway.
This affects only cascading-local, other backends keep using the same Fork mechanisms as they used to before.