[SPARK-10339] [SPARK-10334] [SPARK-10301] [SQL]Partitioned table scan can OOM driver and throw a better error message when users need to enable parquet schema merging

[yhuai]

This fixes the problem that scanning partitioned table causes driver have a high memory pressure and takes down the cluster. Also, with this fix, we will be able to correctly show the query plan of a query consuming partitioned tables.

https://issues.apache.org/jira/browse/SPARK-10339
https://issues.apache.org/jira/browse/SPARK-10334

Finally, this PR squeeze in a "quick fix" for SPARK-10301. It is not a real fix, but it just throw a better error message to let user know what to do.

[yhuai]

@liancheng @marmbrus I add the check to make sure the number of fields of a parquet file's struct have the same number of the corresponding struct in the table schema. If this check fails, we will ask users to enable mergeSchema.

[marmbrus]

test this please

[liancheng]

Just a note: this is a quick fix version of #8509.

[SparkQA]

Test build #41770 has finished for PR 8515 at commit b509bee.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41771 has finished for PR 8515 at commit b509bee.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Would be nice to have the following test case in ParquetHadoopFsRelationSuite:

  test("SPARK-10334 Projections and filters should be kept in physical plan") {
    withTempPath { dir =>
      val path = dir.getCanonicalPath

      sqlContext.range(2).select('id as 'a, 'id as 'b).write.partitionBy("b").parquet(path)
      val df = sqlContext.read.parquet(path).filter('a === 0).select('b)
      val physicalPlan = df.queryExecution.executedPlan

      assert(physicalPlan.collect { case p: execution.Project => p }.length === 1)
      assert(physicalPlan.collect { case p: execution.Filter => p }.length === 1)
    }
  }

And probably add [SPARK-10301] in the PR title, since this PR also delivers a quick fix for it.

Otherwise LGTM. Verified locally that filter push-down and column pruning both work properly.

[SparkQA]

Test build #41786 has finished for PR 8515 at commit 8ecd3a6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks, merging to master and branch-1.5.