Skip to content

Commit 166fba0

Browse files
tanelkdongjoon-hyun
authored andcommitted
[SPARK-32688][SQL][TEST] Add special values to LiteralGenerator for float and double
### What changes were proposed in this pull request? The `LiteralGenerator` for float and double datatypes was supposed to yield special values (NaN, +-inf) among others, but the `Gen.chooseNum` method does not yield values that are outside the defined range. The `Gen.chooseNum` for a wide range of floats and doubles does not yield values in the "everyday" range as stated in typelevel/scalacheck#113 . There is an similar class `RandomDataGenerator` that is used in some other tests. Added `-0.0` and `-0.0f` as special values to there too. These changes revealed an inconsistency with the equality check between `-0.0` and `0.0`. ### Why are the changes needed? The `LiteralGenerator` is mostly used in the `checkConsistencyBetweenInterpretedAndCodegen` method in `MathExpressionsSuite`. This change would have caught the bug fixed in apache#29495 . ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Locally reverted apache#29495 and verified that the existing test cases caught the bug. Closes apache#29515 from tanelk/SPARK-32688. Authored-by: Tanel Kiis <[email protected]> Signed-off-by: Takeshi Yamamuro <[email protected]> (cherry picked from commit 6051755) Signed-off-by: Takeshi Yamamuro <[email protected]>
1 parent cbe5f50 commit 166fba0

File tree

2 files changed

+17
-6
lines changed

2 files changed

+17
-6
lines changed

sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -260,10 +260,10 @@ object RandomDataGenerator {
260260
new MathContext(precision)).bigDecimal)
261261
case DoubleType => randomNumeric[Double](
262262
rand, r => longBitsToDouble(r.nextLong()), Seq(Double.MinValue, Double.MinPositiveValue,
263-
Double.MaxValue, Double.PositiveInfinity, Double.NegativeInfinity, Double.NaN, 0.0))
263+
Double.MaxValue, Double.PositiveInfinity, Double.NegativeInfinity, Double.NaN, 0.0, -0.0))
264264
case FloatType => randomNumeric[Float](
265265
rand, r => intBitsToFloat(r.nextInt()), Seq(Float.MinValue, Float.MinPositiveValue,
266-
Float.MaxValue, Float.PositiveInfinity, Float.NegativeInfinity, Float.NaN, 0.0f))
266+
Float.MaxValue, Float.PositiveInfinity, Float.NegativeInfinity, Float.NaN, 0.0f, -0.0f))
267267
case ByteType => randomNumeric[Byte](
268268
rand, _.nextInt().toByte, Seq(Byte.MinValue, Byte.MaxValue, 0.toByte))
269269
case IntegerType => randomNumeric[Int](

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,16 +68,27 @@ object LiteralGenerator {
6868
lazy val longLiteralGen: Gen[Literal] =
6969
for { l <- Arbitrary.arbLong.arbitrary } yield Literal.create(l, LongType)
7070

71+
// The floatLiteralGen and doubleLiteralGen will 50% of the time yield arbitrary values
72+
// and 50% of the time will yield some special values that are more likely to reveal
73+
// corner cases. This behavior is similar to the integral value generators.
7174
lazy val floatLiteralGen: Gen[Literal] =
7275
for {
73-
f <- Gen.chooseNum(Float.MinValue / 2, Float.MaxValue / 2,
74-
Float.NaN, Float.PositiveInfinity, Float.NegativeInfinity)
76+
f <- Gen.oneOf(
77+
Gen.oneOf(
78+
Float.NaN, Float.PositiveInfinity, Float.NegativeInfinity, Float.MinPositiveValue,
79+
Float.MaxValue, -Float.MaxValue, 0.0f, -0.0f, 1.0f, -1.0f),
80+
Arbitrary.arbFloat.arbitrary
81+
)
7582
} yield Literal.create(f, FloatType)
7683

7784
lazy val doubleLiteralGen: Gen[Literal] =
7885
for {
79-
f <- Gen.chooseNum(Double.MinValue / 2, Double.MaxValue / 2,
80-
Double.NaN, Double.PositiveInfinity, Double.NegativeInfinity)
86+
f <- Gen.oneOf(
87+
Gen.oneOf(
88+
Double.NaN, Double.PositiveInfinity, Double.NegativeInfinity, Double.MinPositiveValue,
89+
Double.MaxValue, -Double.MaxValue, 0.0, -0.0, 1.0, -1.0),
90+
Arbitrary.arbDouble.arbitrary
91+
)
8192
} yield Literal.create(f, DoubleType)
8293

8394
// TODO cache the generated data

0 commit comments

Comments
 (0)