[SPARK-11262][ML] Unit test for gradient, loss layers, memory management for multilayer perceptron

[avulanov]

1.Implement LossFunction trait and implement squared error and cross entropy
loss with it
2.Implement unit test for gradient and loss
3.Implement InPlace trait and in-place layer evaluation
4.Refactor interface for ActivationFunction
5.Update of Layer and LayerModel interfaces
6.Fix random weights assignment
7.Implement memory allocation by MLP model instead of individual layers

These features decreased the memory usage and increased flexibility of
internal API.

[SparkQA]

Test build #44163 has finished for PR 9229 at commit d47310b.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * trait LossFunction\n * trait InPlace\n * class SigmoidLayerWithSquaredError extends Layer\n * class SigmoidLayerModelWithSquaredError extends FunctionalLayerModel(new SigmoidFunction)\n * class SoftmaxLayerWithCrossEntropyLoss extends Layer\n * class SoftmaxLayerModelWithCrossEntropyLoss extends LayerModel with LossFunction\n

[avulanov]

@mengxr Could you take a look?

@feynmanliang These changes clash with some of your changes in #8648, sorry, because I simplified the interface of ActivationFunction to Double => Double and forced all computations to be in place. However, your suggestion of UFunc could be very helpful. Can you think of a way rewriting the UniversalFunction object with UFunc so that it does in-place operations? In any case, we have to merge your changes in BreezeUtil.scala either in the original PR or here and improve the scaladoc as you proposed.

[feynmanliang]

4 indent

[feynmanliang]

Made a pass

[avulanov]

@feynmanliang Thank you! I've addressed all comments, except related to the ones vector because it actually depends on the batch size. With regards to UniversalFunction.apply, I could call it explicitly. Does it sound reasonable to you?

[SparkQA]

Test build #44188 has finished for PR 9229 at commit 7864b8f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44190 has finished for PR 9229 at commit 215d249.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44193 has finished for PR 9229 at commit 6f1200c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45311 has finished for PR 9229 at commit 1d29901.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45410 has finished for PR 9229 at commit 129c9aa.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[avulanov]

@mengxr Could you suggest why it continues to fail? Thanks.

[feynmanliang]

@avulanov On this page search for "*** FAILED ***"

[avulanov]

@feynmanliang Thanks for the tip! My unit tests fail. It has a problem with the new RNG in Spark, so I updated the seeds and the asserts. However the fail seems to correspond to the old code: Should converge to a good optimum after the 3rd restart (3rd). The new code has ...after the 5th restart (5th). Could you suggest why it is happening?

[avulanov]

retest this please

[SparkQA]

Test build #45658 has finished for PR 9229 at commit 083a9b3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[avulanov]

retest this please

[SparkQA]

Test build #45680 has finished for PR 9229 at commit 2a6d644.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45698 has finished for PR 9229 at commit e3ca7f1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[avulanov]

@mengxr Could you suggest what would be the best practice to handle zeros in the computation of logarithm of output? https://github.com/avulanov/spark/blob/mlp-refactoring/mllib/src/main/scala/org/apache/spark/ml/ann/LossFunction.scala#L122

[SparkQA]

Test build #53461 has finished for PR 9229 at commit e5bef10.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[avulanov]

@mengxr Rebased to accomodate #9854

[SparkQA]

Test build #54347 has finished for PR 9229 at commit b21b56a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@avulanov This PR still has conflicts with master.

[avulanov]

@mengxr That is a project/MimaExcludes.scala. Apparently, it is modified almost every day in the master branch. Rebased.

[SparkQA]

Test build #54646 has finished for PR 9229 at commit 94dcec0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[avulanov]

Build exception unrelated to PR

[mengxr]

test this please

[SparkQA]

Test build #54671 has finished for PR 9229 at commit 94dcec0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master. Thanks and sorry for the long delay in code review!

[avulanov]

@mengxr @feynmanliang Thank you!