Random behavior when converting uninitialized Array{Bool} entries

[nalimilan]

This issue has been reported against DataArrays (JuliaStats/DataArrays.jl#233). A short reproducer without using any package can be found in this gist: running the last line multiple times gives different results for several entries which should be defined.

Basically, it appears that leaving some entries in a BitArray undefined iterating over a BitArray can lead to an unexpected behaviour when using the BitArray entries in a condition to decide whether to initialize another array's entries or not. Initializing the array fixes the randomness, as does changing the na::BitArray{N} field to an Array{Bool,N}.

This happens on both 0.5 and master, but not on 0.4.7.

Cc: @carlobaldassi

[yuyichao]

similar never initialize the array.

[nalimilan]

Please read this more carefully. All array entries are manually initialized later except for the first one, and yet they are random. (Or I'm missing something obvious...).

[nalimilan]

That said, description wasn't completely accurate, I've tried to improve it.

[yuyichao]

AFAICT res is not initialized.

[nalimilan]

Well, the loop initializes all entries for which na is false, i.e. all entries except for the first one, right?

[yuyichao]

Though the issue does not seem to be related to the fill!, uncommenting it does not fix the issue. It seems to come from the & or !.

[nalimilan]

Funny. I've tried again, and uncommenting that line definitely fixes the randomness for me, even across 1M runs. Though it could still be another random result...

[carlobaldassi]

I tracked down the issue to the code which converts from Vector{Bool} to BitArray, where the random behaviour occurs. Specifically, in the conversion from Bool to UInt64, when the Bool in question is read from an uninizialized array. Here is a gist with a much simplified code which exposes the bug (see the example run at the bottom of the gist.)

Running it repeatedly seems to suggest the following cause: when the first element of the res array is read, since it was never initialized, random data is returned, in this line:

c |= (UInt64(C[ind]) << j) # the result is random here when ind==1

Changing UInt64(C[ind]) to C[ind]?UInt64(1):UInt64(0) seems to fix this issue, for some reason. However, note that even things like @show ind,C[1] or print("C[$ind]=",C[ind]) result in random behaviour, i.e. they often have different results from @show C[1] or @show C.
I guess this is a problem with the compiler being "too smart" (resulting in undefined behaviour).

I could submit the above fix as a bandaid, but it's probably better if some more knowledgeable people (@JeffBezanson, @vtjnash, ...) could fix the root cause.

[mbauman]

More simply:

f(A, i) = UInt(A[i])
A = Array{Bool}(5)

julia> f(A, 1)
0x0000000000000050

julia> UInt(A[1])
0x0000000000000000