Skip to content

[SPARK-10540] [SQL] Ignore HadoopFsRelationTest's "test all data types" if it is too flaky #8705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Conversation

yhuai
Copy link
Contributor

@yhuai yhuai commented Sep 10, 2015

If hadoopFsRelationSuites's "test all data types" is too flaky we can disable it for now.

https://issues.apache.org/jira/browse/SPARK-10540

@SparkQA
Copy link

SparkQA commented Sep 10, 2015

Test build #42284 has finished for PR 8705 at commit 7c4791f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Sep 11, 2015
…s" if it is too flaky

If hadoopFsRelationSuites's "test all data types" is too flaky we can disable it for now.

https://issues.apache.org/jira/browse/SPARK-10540

Author: Yin Huai <[email protected]>

Closes #8705 from yhuai/SPARK-10540-ignore.

(cherry picked from commit 6ce0886)
Signed-off-by: Yin Huai <[email protected]>
@asfgit asfgit closed this in 6ce0886 Sep 11, 2015
@liancheng
Copy link
Contributor

Found two possible sources of flakiness:

  1. For double and float, NaN == NaN is always false

  2. checkAnswer doesn't handle Map[K, V] properly. For example:

    scala> Map(1 -> 2, 2 -> 1).toString
    res0: String = Map(1 -> 2, 2 -> 1)
    
    scala> Map(2 -> 1, 1 -> 2).toString
    res1: String = Map(2 -> 1, 1 -> 2)
    

    We can't rely on toString to compare Map instances.

Need to update checkAnswer to special case NaN and Map[K, V].

@liancheng
Copy link
Contributor

Filed https://issues.apache.org/jira/browse/SPARK-10591 to track this issue.

@yhuai
Copy link
Contributor Author

yhuai commented Sep 14, 2015

For the issue of NaN, looks like there is something wrong because we want to return true when compare two NaN.

ashangit pushed a commit to ashangit/spark that referenced this pull request Oct 19, 2016
…s" if it is too flaky

If hadoopFsRelationSuites's "test all data types" is too flaky we can disable it for now.

https://issues.apache.org/jira/browse/SPARK-10540

Author: Yin Huai <[email protected]>

Closes apache#8705 from yhuai/SPARK-10540-ignore.

(cherry picked from commit 6ce0886)
Signed-off-by: Yin Huai <[email protected]>
(cherry picked from commit 295281f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants