Inconsistency of results when using `isna` and `&`

[joshbode]

For fixed input, I am getting inconsistent results when using the bitwise and operator &, i.e.

using DataArrays
using DataStructures
julia> x = DataVector{Float64}(0.0:0.1:1.0); x[1] = NA;

julia> counter([count(identity, !isna(x) & (x .>= 0.5)) for i in 1:1000])
DataStructures.Accumulator{Int64,Int64} with 5 entries:
  7  => 54
  9  => 43
  10 => 1
  8  => 50
  6  => 852

The answer is expected to be 6 (and only 6!).

Version: Julia 0.5
Platform: OS X

[joshbode]

Workaround: convert result of isna to a DataVector{Bool}

_(x::AbstractDataVector{Bool}) = DataVector{Bool}(!isna(x)) & x

julia> x = DataVector{Float64}(0.0:0.1:1.0); x[1] = NA;
julia> counter([count(identity, _(x .>= 0.5)) for i in 1:1000])
DataStructures.Accumulator{Int64,Int64} with 1 entry:
  6 => 1000

[nalimilan]

Woops, good catch. & is indeed non-deterministic. I can reproduce this by running this several times:

using DataArrays
x = DataVector{Float64}(0.0:0.1:1.0); x[1] = NA;
(x .>= 0.45).data & !isna(x)

What's weird is that running any of these in isolation always gives the same result. So looks like a very deep bug, maybe in Julia or LLVM?

[joshbode]

This works:

@swappable Base.:&(a::DataArray{Bool}, b::BitArray) = DataArray(convert(Array{Bool}, !(!(a.data & b) | !a.data)), a.na & b)

rather than the existing definition in operators.jl:

@swappable Base.:&(a::DataArray{Bool}, b::BitArray) = DataArray(convert(Array{Bool}, a.data & b, a.na & b)

julia> x = BitArray([true, true, false, false, true, false]);
julia> y = DataVector{Bool}([true, false, true, false, true, true]);
julia> y[5:6] = NA;
julia> collect(Any, zip(x, y, x & y))
6-element Array{Any,1}:
 (true,true,true)
 (true,false,false)
 (false,true,false)
 (false,false,false)
 (true,NA,NA)
 (false,NA,false)

There's probably a more efficient way to express this, though.

[nalimilan]

But there's clearly a bug somewhere that we should find as it's quite dangerous. Ideally we would find a reproducer without loading DataArrays, e.g. by copying only its type definition. Then we could file it against Base Julia.

[joshbode]

Hmm - it seems that the presence of at least one NA in a vector causes randomness in the data field of the DataVector{Bool} result of the comparison:

julia> x = DataVector{Float64}(0.0:0.1:1.0);
julia> counter([(x .>= 0.5).data for i in 1:1000])
DataStructures.Accumulator{Array{Bool,1},Int64} with 1 entry:
  Bool[false,false,false,false,false,true,true,true,true,true,true] => 1000

julia> x[1] = NA
NA

julia> counter([(x .>= 0.5).data for i in 1:1000])
DataStructures.Accumulator{Array{Bool,1},Int64} with 2 entries:
  Bool[false,false,false,false,false,true,true,true,true,true,true] => 937
  Bool[true,false,false,false,false,true,true,true,true,true,true]  => 63

Combined with the definition for & for a DataVector and a BitArray, the randomly allocated data in the NA slots of the DataVector is getting picked up.

[joshbode]

Simplest I could get to is this, which seems to indicate the Julia base Vector{Bool} to BitVector conversion isn't working:

julia> x = DataVector{Int64}(1:3); x[1] = NA;
julia> counter([(a = (x .== 0.0).data; (a[2:end], convert(BitVector, a))) for i in 1:1000])
DataStructures.Accumulator{Tuple{Array{Bool,1},BitArray{1}},Int64} with 6 entries:
  (Bool[false,false],Bool[false,true,true])   => 2
  (Bool[false,false],Bool[false,false,false]) => 616
  (Bool[false,false],Bool[true,true,false])   => 6
  (Bool[false,false],Bool[false,true,false])  => 6
  (Bool[false,false],Bool[true,false,false])  => 369
  (Bool[false,false],Bool[true,true,true])    => 1

However, I can't find anything that doesn't involve a DataVector in some way (if only through the data field). Note: even if a is replaced with copy(a) the issue is still present.

[nalimilan]

Of course it's expected that for NA entries the underlying value is undefined. But here the problem is different, since the variations in the code I posted above also affect other entries than the first one. See the result of:

counter([(x .>= 0.45).data & !isna(x) for _ in 1:1000])

I've been able to create a simple reproducer: https://gist.github.com/nalimilan/30d87f1e810afce76660e79ab5dc22f7 The same behavior appears on Julia 0.6. Filling the bit array with true or false fixes the bug, which seems to indicate there's indeed a bug in Julia.

[nalimilan]

OK, filed as JuliaLang/julia#20065.

[joshbode]

Ta - thanks for that :)

[nalimilan]

Reported again in this thread.

[joshbode]

Closing this.
Just tested this on 0.6 and the result is now as expected:

using DataArrays
using DataStructures
julia> x = DataVector{Float64}(0.0:0.1:1.0); x[1] = NA;
julia> counter([count(identity, .!isna.(x) .& (x .>= 0.5)) for i in 1:1000])
DataStructures.Accumulator{Int64,Int64} with 1 entry:
  6 => 1000