Skip to content

Commit d38887b

Browse files
Li Pumengxr
authored andcommitted
use specialized axpy in RowMatrix for SVD
After running some more tests on large matrix, found that the BV axpy (breeze/linalg/Vector.scala, axpy) is slower than the BSV axpy (breeze/linalg/operators/SparseVectorOps.scala, sv_dv_axpy), 8s v.s. 2s for each multiplication. The BV axpy operates on an iterator while BSV axpy directly operates on the underlying array. I think the overhead comes from creating the iterator (with a zip) and advancing the pointers. Author: Li Pu <[email protected]> Author: Xiangrui Meng <[email protected]> Author: Li Pu <[email protected]> Closes apache#1378 from vrilleup/master and squashes the following commits: 6fb01a3 [Li Pu] use specialized axpy in RowMatrix 5255f2a [Li Pu] Merge remote-tracking branch 'upstream/master' 7312ec1 [Li Pu] very minor comment fix 4c618e9 [Li Pu] Merge pull request #1 from mengxr/vrilleup-master a461082 [Xiangrui Meng] make superscript show up correctly in doc 861ec48 [Xiangrui Meng] simplify axpy 62969fa [Xiangrui Meng] use BDV directly in symmetricEigs change the computation mode to local-svd, local-eigs, and dist-eigs update tests and docs c273771 [Li Pu] automatically determine SVD compute mode and parameters 7148426 [Li Pu] improve RowMatrix multiply 5543cce [Li Pu] improve svd api 819824b [Li Pu] add flag for dense svd or sparse svd eb15100 [Li Pu] fix binary compatibility 4c7aec3 [Li Pu] improve comments e7850ed [Li Pu] use aggregate and axpy 827411b [Li Pu] fix EOF new line 9c80515 [Li Pu] use non-sparse implementation when k = n fe983b0 [Li Pu] improve scala style 96d2ecb [Li Pu] improve eigenvalue sorting e1db950 [Li Pu] SPARK-1782: svd for sparse matrix using ARPACK
1 parent 5596086 commit d38887b

File tree

1 file changed

+7
-1
lines changed
  • mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed

1 file changed

+7
-1
lines changed

mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,13 @@ class RowMatrix(
8383
seqOp = (U, r) => {
8484
val rBrz = r.toBreeze
8585
val a = rBrz.dot(vbr.value)
86-
brzAxpy(a, rBrz, U.asInstanceOf[BV[Double]])
86+
rBrz match {
87+
// use specialized axpy for better performance
88+
case _: BDV[_] => brzAxpy(a, rBrz.asInstanceOf[BDV[Double]], U)
89+
case _: BSV[_] => brzAxpy(a, rBrz.asInstanceOf[BSV[Double]], U)
90+
case _ => throw new UnsupportedOperationException(
91+
s"Do not support vector operation from type ${rBrz.getClass.getName}.")
92+
}
8793
U
8894
},
8995
combOp = (U1, U2) => U1 += U2

0 commit comments

Comments
 (0)