[SVM] LayoutEncoding.getArrayBaseOffset() causes assertion failure

[y1yang0]

We found the implementation of layout encoding and decoding is asymmetric:

// encoding
public static int forArray(boolean isObject, int arrayBaseOffset, int arrayIndexShift) {
    int tag = isObject ? ARRAY_TAG_OBJECT_VALUE : ARRAY_TAG_PRIMITIVE_VALUE;
    int encoding = (tag << ARRAY_TAG_SHIFT) | (arrayBaseOffset << ARRAY_BASE_SHIFT) | (arrayIndexShift << ARRAY_INDEX_SHIFT_SHIFT);
    // assertions
    assert !isInstance(encoding) && isArray(encoding);
    assert isObjectArray(encoding) == isObject;
    assert isPrimitiveArray(encoding) != isObject;
    assert getArrayBaseOffset(encoding).equal(WordFactory.unsigned(arrayBaseOffset));
    ^^^^^^^^^^^^^^^^^^^^^^^^^^ throw AssertionError
    assert getArrayIndexShift(encoding) == arrayIndexShift;
    return encoding;
}
// decoding
public static UnsignedWord getArrayBaseOffset(int encoding) {
    return WordFactory.unsigned((encoding >> ARRAY_BASE_SHIFT) & ARRAY_BASE_MASK);
    ^^^^^^^^^^^^^^^^
}

In encoding method, arrayBaseOffset has 22 bits which range from 8th to 29th inclusively, but in decoding method, arrayBaseOffset only contains 8 bits because of array base mask (255), this
would cause a potential problem if array base offset excesses 255, we can reproduce this scenario by the following case:

import sys

TMPL = "public class C{} implements I{}, I{}, I{} {{}}"
TMPL2 = "public interface I{} {{}}"

if __name__ == "__main__":

    N = 1000
    for i in range(0,N):
        with open('C{}.java'.format(i), 'w') as f:
            f.write(TMPL.format(i,i,((i + 3) % N), ((i+5) % N)))
        with open('I{}.java'.format(i), 'w') as f:
            f.write(TMPL2.format(i))

    with open('Main.java', 'w') as f:
        f.write("public class Main {")
        f.write("public static void main(String[] args) {")
        f.write("int hash = 0;");
        for i in range(0,N):
            f.write("C{} c{} = new C{}();".format(i, i, i));
            f.write("hash ^= c{}.hashCode();".format(i));
        f.write("System.exit(hash);}}")

Commands:

python gen.py
javac Main.java
native-image -J-ea Main

This script will generate many interfaces and classes, and lead to big interface-bits.
​native-image will throw an AssertError due to the previous problem. Is this a bug or an intentional design of native-image?
Can we change the ARRAY_BASE_MASK from 255 to (1 << (Integer.SIZE - ARRAY_TAG_BITS - ARRAY_BASE_SHIFT)) - 1 ？

[peter-hofer]

Thanks for your report @kelthuzadx ! Fixed by adding extra bits in bbfd632

[y1yang0]

@peter-hofer This patch changes ARRAY_BASE_MASK to 0xfff and use VMError.guarantee to guarantee a valid value. But it does not solve the root cause of the problem: encoding layout_helper remains 22 bits for array base offset, decoding layout_helper uses just 12 bits(0xfff). Why we use 0xfff as its mask rather than using 0x3fffff(22bits)?

[peter-hofer]

12 bits allow for an offset of up to 4095, which should be sufficient in practice. The root cause of this problem is the hybrid layout of DynamicHub that contains a bitset for instanceof checks which can grow arbitrarily large depending on the class hierarchy and pushes the array further back in the layout, increasing its offset beyond the bounds of LayoutEncoding. Fixing this underlying issue will be more complex.