Fix/merge idx

[s-kuberski]

This pr fixes two bugs:

1. When merging observables that are defined on different regular sets of configurations (i.e., the idls being ranges), the old version of the code made an attempt to merge those smartly. However, the corresponding piece of code did not take into account all possible combinations, such that merging range(1, 1011, 2) and range(1, 1010, 1) led to range(1, 1011, 1), clearly adding a configuration 1010 that was not present before.

   The merge of idls is now always performed via sorted(set().union(*idl)) when the sets of configurations differ. Afterwards, an attempt is made to check whether the resulting union can be expressed as range. The resulting overhead of this check is much smaller than the total time that is needed to merge observables that are based on different sets of configurations. Using union certainly adds extra time compare to the previous (wrong) solution of merging, but I failed to find an analytical solution to either determine the range that describes the union of ranges or returns the list that describes the union in the most general cases. For the merge of matching idl, nothing has changed.

   A similar change has been applied to _intersection_idx to have the same, correct, definition of finding the union and intersection of a list of idls.

   Some tests that had been written to check correctness have been wrong themselves. I corrected them and added a few more.

2. The second fix concerns an edge case where the solution of _calculate_drho for large time separations #157 did lead to an exception, since the list in _calculate_drho was missing an element.

[fjosw]

Thanks, that looks good, I will have a closer look later.

One thing that I noticed is the use of the generic except Exception which could potentially also catch nonintended exceptions. Could you replace it with the exception type that is thrown if the elements are not identical? (I was not very careful with these things in the past but try to improve it along the way)

[s-kuberski]

Good point. I'll try to catch the relevant exceptions.

[fjosw]

A more cosmetic thing that I noticed is that we reuse these groupby patterns a few times. We could consider refactoring them.

[s-kuberski]

Is it possible that you introduced the try...except because np.ndarrays would otherwise break the workflow because they are not iterable? I did not find any other case where, given that we only pass lists of lists, ranges and arrays, an exception would be thrown. The more sane thing would then to introduce something like

idl = [np.nditer(o) if isinstance(o, np.ndarray) else o for o in idl]

and to remove the try altogether.

I will think how to refactor the pattern.

[fjosw]

That might be the case. The other thing I could imagine are empty lists for which you would get an IndexError but that should be excluded anyways.

[s-kuberski]

I guess we would want an exception to be thrown in the case where we pass an empty list ;)

[fjosw]

The CI fails because of wrong dependencies in the latest IPython release (ipython/ipython#14053). I will still merge this and hope that they resolve the issue soon.