[SPARK-9471] [ML] Multilayer Perceptron #7621

avulanov · 2015-07-23T15:58:16Z

Summary

This pull request contains the following feature for ML:

Multilayer Perceptron classifier

This implementation is based on our initial pull request with @bgreeven: #1290 and inspired by very insightful suggestions from @mengxr and @witgo (I would like to thank all other people from the mentioned thread for useful discussions). The original code was extensively tested and benchmarked. Since then, I've addressed two main requirements that prevented the code from merging into the main branch:

Extensible interface, so it will be easy to implement new types of networks
- Main building blocks are traits Layer and LayerModel. They are used for constructing layers of ANN. New layers can be added by extending the Layer and LayerModel traits. These traits are private in this release in order to save path to improve them based on community feedback
- Back propagation is implemented in general form, so there is no need to change it (optimization algorithm) when new layers are implemented
Speed and scalability: this implementation has to be comparable in terms of speed to the state of the art single node implementations.
- The developed benchmark for large ANN shows that the proposed code is on par with C++ CPU implementation and scales nicely with the number of workers. Details can be found here: https://github.com/avulanov/ann-benchmark

Other implementations based on the proposed interface

@mengxr and @dbtsai kindly agreed to perform code review.

ANN test

SparkQA · 2015-07-23T16:08:03Z

Test build #38237 has finished for PR 7621 at commit a226133.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier (override val uid: String)
- class MultilayerPerceptronRegressor (override val uid: String)
- class FeedForwardTrainer (topology: Topology, val inputSize: Int,

SparkQA · 2015-07-23T16:21:10Z

Test build #38239 has finished for PR 7621 at commit e191301.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier (override val uid: String)
- class MultilayerPerceptronRegressor (override val uid: String)
- class FeedForwardTrainer (topology: Topology, val inputSize: Int,

SparkQA · 2015-07-23T17:03:58Z

Test build #38242 has finished for PR 7621 at commit 35125ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier (override val uid: String)
- class MultilayerPerceptronRegressor (override val uid: String)
- class FeedForwardTrainer (topology: Topology, val inputSize: Int,
- trait ExpectsInputTypes extends Expression
- trait ImplicitCastInputTypes extends ExpectsInputTypes
- trait Unevaluable extends Expression
- trait Nondeterministic extends Expression
- trait CodegenFallback extends Expression
- case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

witgo · 2015-07-24T02:05:52Z

mllib/src/main/scala/org/apache/spark/mllib/ann/Layer.scala

+ * Implements Layer instantiation.
+ *
+ */
+private[ann] trait Layer extends Serializable {


I think it should be public. Some people may want to customize it.

@mengxr suggested to make it private in this release to be able to modify it if needed and make public in the next one.

OK, I see. This is understandable.

I'm not sure I understand the benefit of separating the Layer and LayerModel this way. Can we have a unified Layer, just by moving the getInstance methods to LayerModel and rename it to Layer?

We need a separate lightweight instance for holding the Layer properties, so we can pass it easily through the network to executors on each iteration. We don't want to pass LayerModels because they contain weights and weights are transmitted by the GradientDescent or LBFGS broadcast/treeAggregate routines. Does it answer your question?

witgo · 2015-07-24T15:32:01Z

mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala

+                                                      layers: Array[Int],
+                                                      weights: Vector)
+  extends PredictionModel[Vector, MultilayerPerceptronClassifierModel]
+  with Serializable {


Support model save/load?

Do you know if there exist a generic model loader/saver in Spark ML? I can think only about using sc.parallelize(Seq(model), 1).saveAsObjectFile("model") that does not look good, honestly speaking.

Let's do that in a follow-up PR to keep this PR minimal.

SparkQA · 2015-07-24T16:06:17Z

Test build #38358 has finished for PR 7621 at commit 9d18469.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier (override val uid: String)
- class MultilayerPerceptronRegressor (override val uid: String)
- class FeedForwardTrainer (topology: Topology, val inputSize: Int,

hhbyyh · 2015-07-28T14:43:34Z

mllib/src/main/scala/org/apache/spark/mllib/ann/Layer.scala

+   * @return delta
+   */
+  def prevDelta(nextDelta: BDM[Double], input: BDM[Double]): BDM[Double]
+


It's possible that prevDelta needs to get some properties from the next layer. (at least for CNN). I'm not sure if it's a special (rare) requirement. Of course there's some workaround, yet it will be handy if there's a reference to the next layer.

Could you elaborate on which properties are needed from the next layer? @witgo pointed that weights initialization in AffineLayer depends on the next FunctionalLayer layer https://github.com/apache/spark/pull/7621/files#r35435404. The latter can be handled in the ANN fabric.

the number of feature maps in the next layer

mengxr · 2015-07-28T18:53:03Z

@avulanov Could you update the PR title to be more specific, e.g., "[SPARK-2352][ML] Multilayer Perceptron"?

mengxr · 2015-07-28T19:24:51Z

mllib/src/main/scala/org/apache/spark/ml/regression/MultilayerPerceptronRegressor.scala

+  with Logging {
+
+  /** @group setParam */
+  def setInputCol(value: String): this.type = set(inputCol, value)


Do we want to use inputCol and outputCol instead of featuresCol and labelCol?

SparkQA · 2015-07-30T11:07:23Z

Test build #39037 has finished for PR 7621 at commit f69bb3d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier(override val uid: String)

…d DataStacker class

avulanov · 2015-07-30T11:27:11Z

@mengxr Thank you for your review! All comments are addressed. Should I change the name of my git branch (it is SPARK-2352-ann)?

mengxr · 2015-07-30T15:03:58Z

Thanks! The branch name doesn't matter:) I will make another pass today.

SparkQA · 2015-07-30T15:46:31Z

Test build #39045 has finished for PR 7621 at commit a7e7951.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier(override val uid: String)

witgo · 2015-07-30T17:02:25Z

LGTM

mengxr · 2015-07-30T22:29:07Z

mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala

+   */
+  def setSeed(value: Long): this.type = set(seed, value)
+
+  setDefault(maxIter -> 100, tol -> 1e-4, layers -> Array(1, 1), blockSize -> 100)


Remove default value for layers. Shall we change default block size to a power of 2, e.g., 128?

mengxr · 2015-07-30T22:31:01Z

@avulanov I haven't finished a full pass. But feel free to update the PR to address current comments.

avulanov · 2015-07-31T00:05:17Z

@mengxr Thanks again! Done.

SparkQA · 2015-07-31T00:45:55Z

Test build #39126 has finished for PR 7621 at commit 4806b6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier(override val uid: String)

mengxr · 2015-07-31T18:28:51Z

LGTM. Merged into master. There are some minor issues we can address during the QA period. Thanks @avulanov for the implementation and performance tests, and everyone who helped!!

sbrouil · 2016-04-13T21:53:32Z

@avulanov There is no way to access probabilities of each class on the MultilayerPerceptronClassificationModel before the predict method decode it as a label ? It would be interesting to have such feature but instead the model is a blackbox with everything private. Any reason for that ?

avulanov · 2016-04-13T22:17:24Z

@sbrouil Indeed, it worth implementing this feature. We have already discussed it in the mailing list https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3C9D5B00849D2CDA4386BDA89E83F69E6C194CF351@G9W0737.americas.hpqcorp.net%3E

Multilayer Perceptron regressor and classifier

a226133

ANN test

Apache header

e191301

Style fix in tests

35125ab

avulanov mentioned this pull request Jul 23, 2015

[SPARK-9273] [MLlib] Add Convolutional Neural network to Spark MLlib #7609

Closed

witgo reviewed Jul 24, 2015
View reviewed changes

Addressing reviewers comments: unnecessary copy of data in predict

9d18469

witgo reviewed Jul 24, 2015
View reviewed changes

hhbyyh reviewed Jul 28, 2015
View reviewed changes

mengxr reviewed Jul 28, 2015
View reviewed changes

Addressing reviewers comments.

f69bb3d

avulanov changed the title ~~[SPARK-2352] [ML] Multilayer Perceptron~~ [SPARK-9471] [ML] Multilayer Perceptron Jul 30, 2015

Default blockSize: 100. Added documentation to blockSize parameter an…

a7e7951

…d DataStacker class

mengxr reviewed Jul 30, 2015
View reviewed changes

Addressing reviewers comments.

4806b6f

asfgit closed this in 6add4ed Jul 31, 2015

mengxr mentioned this pull request Jul 31, 2015

[MLLIB] [spark-2352] Implementation of an Artificial Neural Network (ANN) #1290

Closed

[SPARK-9471] [ML] Multilayer Perceptron #7621

[SPARK-9471] [ML] Multilayer Perceptron #7621

Uh oh!

Conversation

avulanov commented Jul 23, 2015

Summary

Other implementations based on the proposed interface

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mengxr commented Jul 28, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

avulanov commented Jul 30, 2015

Uh oh!

mengxr commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

witgo commented Jul 30, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mengxr commented Jul 30, 2015

Uh oh!

avulanov commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

mengxr commented Jul 31, 2015

Uh oh!

sbrouil commented Apr 13, 2016

Uh oh!

avulanov commented Apr 13, 2016

Uh oh!

Uh oh!