Add optimal "numberOfTrailingZeros" method

[jtmcdole]

#37789 Did a lot of speed up NTZ, but it was suggested that this basic operation could/should be provided by the SDK as the most optimal for the given platform. Example GCC:

Built-in Function: int __builtin_ctz (unsigned int x)
Returns the number of trailing 0-bits in x, starting at the least significant bit position. If x is 0, the result is undefined.

[aartbik]

@mraleph @mkustermann @a-siva what is the proper procedure to add methods to the library, do we have a review committee for that?

The steps to be taken

• add leading/trailing zeros to library, perhaps popcount too while we are at it, using Dart implementations as base functionality
• recognize these method in the DartVM/compiler, lower these to the best possible instruction on each given architecture

[mraleph]

do we have a review committee for that?

Lasse (@lrhn) owns core libraries, so he decides where things should go.

[lrhn]

Adding methods to int is currently easy - since you can't extend or implement int, it won't break anyone. (That won't stay true after adding extension methods).

If I read the GCC docs correctly, "trailing zeros" are the low bits (not at all obvious if you don't know it already, especially when most platforms are little-endian).
We already have bitLength which is equivalent to the high zero bits for positive numbers, so there is precedence for such operations.
I assume we'd want both leading and trailing zero counts, and maybe even popcount. Do we want to count leading/trailing ones?

It's not clear what the JS compiled version of this would be, but counting low zero bits on doubles is definitely possible (negative numbers may need twiddling), but more likely they will just convert to 32-bit integers before doing any bitwise operations.

Now, names are hard.
Maybe int get lowZeroBitCount, int get highZeroBitCount, int get nonZeroBitCount? I want bit in there, I personally do not like "leading" and "trailing", but I do recognize that they are considered standard (writing the digits as a normal base-2 number, so we can define clz as (n.toRadixString(2) + "1").indexOf("1") if necessary). If we use them then countLeadingZeroBits, countTrailingZeroBits, countNonZeroBits.

Does it matter that our numbers are signed (it doesn't affect the bit patterns, but will it perhaps confuse users that -1 has no leading zeros/)

So, we can add these operations. If there is sufficient support for doing so (and people willing to do the implementation on all platforms), then I'm fine with doing it.

[jtmcdole]

+1 for "{low|high}ZeroBitCount" .

Not sure how I feel about signed numbers, but if you look at the quoted issue (now closed); you can do ntz with a const array, you'd just want to make it work for 2^53?

https://gist.github.com/jtmcdole/297434f327077dbfe5fb19da3b4ef5be

The fixnum package already had some API as well:

https://pub.dev/documentation/fixnum/latest/fixnum/Int64/numberOfTrailingZeros.html

[aartbik]

The difference between signed and unsigned is a bit moot for the DartVM, since the operations are purely defined in terms of the bits in the 64-bit int (with possible two's complement interpretation). Big or little endian does not play a role at all here, since that only relates to the order in which the individual bytes appear to the outside world (memory, network, etc). The bits in a 64-bit int are well-defined from lsb to msb on all architectures.

Or, if my words are unclear, let my table do the talking :-)

Below I generated a table with all interesting bit patterns (in hex), the unsigned and signed interpretation of these bits, and the expected values of NTZ, NLZ, and popcount.

I think the only bike shedding we can do on the int behavior is what should be returned for the corner case (0x0000000000000000) for NTZ and NLZ (all zeros).

Bit pattern (hex)   Unsigned             Signed               NTZ NLZ Popcount
------------------------------------------------------------------------------
0x8000000000000000  9223372036854775808 -9223372036854775808   63    0    1
0x8000000000000001  9223372036854775809 -9223372036854775807    0    0    2
0x8000000000000002  9223372036854775810 -9223372036854775806    1    0    2
0x8000000000000003  9223372036854775811 -9223372036854775805    0    0    3
0x8000000000000004  9223372036854775812 -9223372036854775804    2    0    2
0x8000000000000005  9223372036854775813 -9223372036854775803    0    0    3
....
0xfffffffffffffffb 18446744073709551611                   -5    0    0   63
0xfffffffffffffffc 18446744073709551612                   -4    2    0   62
0xfffffffffffffffd 18446744073709551613                   -3    0    0   63
0xfffffffffffffffe 18446744073709551614                   -2    1    0   63
0xffffffffffffffff 18446744073709551615                   -1    0    0   64
0x0000000000000000                    0                    0  <bike-shed> 0
0x0000000000000001                    1                    1    0   63    1
0x0000000000000002                    2                    2    1   62    1
0x0000000000000003                    3                    3    0   62    2
0x0000000000000004                    4                    4    2   61    1
0x0000000000000005                    5                    5    0   61    2
....
0x7ffffffffffffffa  9223372036854775802  9223372036854775802    1    1   61
0x7ffffffffffffffb  9223372036854775803  9223372036854775803    0    1   62
0x7ffffffffffffffc  9223372036854775804  9223372036854775804    2    1   61
0x7ffffffffffffffd  9223372036854775805  9223372036854775805    0    1   62
0x7ffffffffffffffe  9223372036854775806  9223372036854775806    1    1   62
0x7fffffffffffffff  9223372036854775807  9223372036854775807    0    1   63

[aartbik]

I am okay with lowZeroBitCount/highZeroBitCount/nonZeroBitCount. Slightly non-standard nomenclature, but that is perhaps the Dart way :-) If we want to be more conventional, numberOfTrailingZeros/numberOfLeadingZeros would be the choice.

Also, no bike shedding the NTZ/NTL(0)? My suggestions is to keep it simple and return 64 in both cases.....

Also, I am mainly interested in adding this to int (not to num or double of course). Also, I am willing to do the implementation, provided I understand correctly what that involves at the library side (which is still a bit new to me). The DartVM work is certainly on me!

[rakudrama]

I'd prefer we didn't add methods to int that are antagonistic to the web platform.

count-leading-zeros assumes a word length.

int.bitLength has existed for some time and is a word-length agnostic method related to counting leading zeros or ones (depending on what the most-leading, or sign, bit is).
Being word-length agnostic, it tends to 'keep working' better when the code is run compiled to JavaScript or the code is converted to BigInt, both of which have the property that there is no clear word length.

Would it be possible to better optimize the existing int.bitLength and derive the methods like clz64() and clz32() from that, implemented as extension methods or methods in dart:math? If the machine has a clz64 instruction, int.bitLength would be implemented in terms of that instruction and subtraction, so perhaps the compiler could recognize that 64-(64-x) --> x for fixed-word-length arithmetic.

[lrhn]

I've assume that the JS operations would be bitwise operations and convert to Int32 before counting.
That would mean that 0.highZeroBits returns 32 in JavaScript and 64 in native code. That's consistent and compatible with the other bitwise operations in JavaScript (|, &, ^, ~), but not with bitLength.

The problem with using the JS double's representation for anything is that most of the operations gives different results even for numbers that are (negative) 32 bit integers.

We might be able to find a reasonable compromise that still gives the VM a useful result, which also allowing dart2js to not truncate to 32 bits, but I'm not sure it's worth it. Truncating is probably better (or at least more consistent, I'm not particularly fond of of that discrepancy to begin with).

Leading zeros is the only one which assumes a finite length for all operands (trailing zeros only does so for 0). It is definable in terms of bitLength (x < 0 ? 0 : 64 - x.bitLength) so omitting it is not impossible. We can also do it as countHighZeroBits([int bitSize]) which first truncates to toSigned(bitSize) and then counts relative to that, or countHighZeroBits32 and countHighZeroBits64 both being available (although the 32 looks spurious).

The population count can work for positive doubles, but for negative values the result is different from the same on the VM if we just count bits in mantissa. We can flip the result for negative numbers, but only if we know the bit-size, so again, maybe: countNonZeroBits([int bitSize]) which truncates/sign-extends to that number of bits and counts the ones.

That leaves the low zero bits. Converting to a finite width isn't as useful here, this is more like bitLength. Also, for 2's complement numbers, it's symmetric around zero (-2 and 2 are both even numbers), so you can ignore the sign. We still need a value for zero.
If we define it as returning the position of the least signficant non-zero bit (where bitLength returns the most significant one) for positive numbes, and -1 for zero, then JS can answer this truthfully for doubles as well.
This effectively sign/zero-extends the number to "infinite precision two's complement" and gives the position of the first 1 bit (if any).

Just truncating to 32-bit operations for JS is easier. It gives different results for counting leading zeros, but I think that's as acceptable as what we do for and/or/xor.