[SPARK-8707] RDD#toDebugString fails if any cached RDD has invalid partitions

[navis]

Added numPartitions(evaluate: Boolean) to RDD. With "evaluate=true" the method is same with "partitions.length". With "evaluate=false", it checks checked-out or already evaluated partitions in the RDD to get number of partition. If it's not those cases, returns -1. RDDInfo.partitionNum calls numPartition only when it's accessed.

[kmadhugit]

We may not need to introduce another version of toDebugString with a argument to get around this issue. The reported problem is - if there are any unreleated invalid RDDs in the same application it tries to evaluate all of them unnecessarily and fails. So we should be restricting toDebugString evaluate its own partitions.

[andrewor14]

add to whitelist

[andrewor14]

retest this please

[SparkQA]

Test build #36350 has finished for PR 7127 at commit acf3661.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[navis]

Adressed comments. Thanks, @kmadhugit

[SparkQA]

Test build #36539 has finished for PR 7127 at commit 966bcd9.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Hex(child: Expression) extends UnaryExpression with ExpectsInputTypes
  • case class UnHex(child: Expression) extends UnaryExpression with ExpectsInputTypes
  • case class ShiftLeft(left: Expression, right: Expression)
  • case class ShiftRight(left: Expression, right: Expression)
  • case class ShiftRightUnsigned(left: Expression, right: Expression)
  • case class Levenshtein(left: Expression, right: Expression) extends BinaryExpression

[SparkQA]

Test build #41341 has finished for PR 7127 at commit 6698646.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41580 has finished for PR 7127 at commit c61b350.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

I would just make this private[spark]. Let's try to limit the number of things we expose.

[andrewor14]

LGTM, left a few minor comments. I'll merge this once you address them.

[andrewor14]

retest this please

[SparkQA]

Test build #41888 has finished for PR 7127 at commit c61b350.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

I've merged this into master after applying the change myself