attempting to compare a cf field with None returns an error

[bnlawrence]

I have some code which has a comparator that looks like

x = None
loop over stuff
    if x == None:
       # create a cf field called x
   use x in each instance of the loop thereafter.

(I actually do this with if (x,y) == (None,None) but I think it's the simpler problem. My error message is this:

Traceback (most recent call last):
  File "/home/users/lawrence/GitRepos/cmip-play/regrid.py", line 71, in <module>
    if (land, sea) ==  (None, None):
  File "/home/users/lawrence/miniconda3v/envs/py39n/lib/python3.9/site-packages/cf/mixin/propertiesdata.py", line 342, in __eq__
    return self._binary_operation(y, "__eq__")
  File "/home/users/lawrence/miniconda3v/envs/py39n/lib/python3.9/site-packages/cf/field.py", line 1918, in _binary_operation
    if other.ndim > 0:
  File "/home/users/lawrence/miniconda3v/envs/py39n/lib/python3.9/site-packages/cf/data/data.py", line 7679, in ndim
    return self._ndim
  File "/home/users/lawrence/miniconda3v/envs/py39n/lib/python3.9/site-packages/cf/data/data.py", line 7256, in _ndim
    return self._custom["_ndim"]
KeyError: '_ndim'

It looks like the eq method needs to start by seeing if the comparator is of the same class before testing attributes.

I would expect this to work. Clearly it's easy to work around, but ...

cf.environment(paths=False)
Platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.17
HDF5 library: 1.10.6
netcdf library: 4.7.4
udunits2 library: /home/users/lawrence/miniconda3v/envs/py39n/lib/libudunits2.so.0
Python: 3.9.7
netCDF4: 1.5.6
cftime: 1.5.1.1
numpy: 1.21.2
psutil: 5.8.0
scipy: 1.5.3
matplotlib: 3.4.3
ESMF: 8.0.1
cfdm: 1.9.0.1
cfunits: 3.3.4
cfplot: 3.1.17
cf: 3.11.0

[davidhassell]

Hi Bryan, the binary operators on Data don't insist on the other operand being a cf-defined object, so this works:

>>> d = cf.Data(np.array([1, 2, 3]))
>>> d == 1
<CF Data(3): [True, False, False]>

It does this by casting the RHS (in the case) to a Data object, i.e. cf.Data.asdata(<RHS>). This is fine when the RHS is, say, a number, but when it is None the initialization is equivalent to cf.Data.asdata() and just makes an empty shell of a Data object that isn't much good for anything. This is a special case that can easily be test for, and I'll put in a PR for it.

Beware, though, the result would not be a boolean, but a cf.Data object of booleans with the same same shape as the LHS, i.e. the same as this (which works today):

>>> cf.Data(np.array([1, 2, 3]))  == cf.Data([None])
<CF Data(3): [False, False, False]>

which is exactly how numpy would do it:

>>> np.array([1, 2, 3]) == None
array([False, False, False])

[bnlawrence]

Fair point, but CF should then return an appropriate error in the example I used above, which it does not. E.g. this from numpy:

x = numpy.array([1,2,3])
if x != None:
   print('nope')

returns

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

[bnlawrence]

To be honest, I think that numpy is broken here too. If I am testing for "==" if the RHS is a different type of object than the LHS, the equality is definitely NOT true, these more sophisticated issues should not be in play.

[davidhassell]

Hi Bryan, I realise that perhaps is None (or is not None) is what you're after:

x = None
loop over stuff
    if x is None:
       # create a cf field called x
   use x in each instance of the loop thereafter.

That will work. Similarly

x = numpy.array([1,2,3])
if x is None:
   print('nope')

I don't think that Python mandates that the == operator (delivered by __eq__()) must return a boolean. From the Python data model on the rich comparison operators (==, <, >=, etc.) we have

By convention, False and True are returned for a successful comparison. However, these methods can return any value, so if the comparison operator is used in a Boolean context (e.g., in the condition of an if statement), Python will call bool() on the value to determine if the result is true or false.

Having element-wise comparisons array is very useful, in general.

Thanks, David

[bnlawrence]

I'm not disagreeing with the value of elementwise comparison, I'm saying that if A is box of fruit and B is a tray of cookies then if you do A == B one doesn't have to compare the contents of the containers to know that the right answer is False.

However, I agree that for my use case it would most tidily be achieved by is None.

In any case, I'm glad you're putting in a PR so the right error is raised rather than a stackdump. The rest of this is rather esoteric and can wait til we have a suitable pin and more angelic aspects.

[davidhassell]

+1. Looking forward to the philosophical debate!

[sadielbartholomew]

@bnlawrence after I merged the accompanying PR our linting tool (flake8) reports, referencing line d = cf.Data([1, 2, 3]) == None:

cf/test/test_Data.py:1925:32: E711 comparison to None should be 'if cond is None:'

(though I'll add a skip in for that lint warning so it is not raised, as we are of course aware about that). PEP8 ultimately says that isn't the right comparison, though I agree that we should provide the same error message as NumPy, as the PR fix has implemented.