Integrate with ONNX 1.16.0 release branch

[cjvolzka]

We are releasing ONNX 1.16.0. A release branch is created (https://github.com/onnx/onnx/tree/rel-1.16.0). The planned release date is March 25, 2024. Release candidates are also available from TestPyPI: pip install -i https://test.pypi.org/simple/ --pre onnx

It is important to integrate ONNX release branch ASAP so that any issues and incompatibilities can be detected and resolved before the ONNX release.

Key updates:

• ai.onnx Opset 21
  • Update to support int4 and uint4:
    • Cast, CastLike, Constant, ConstantOfShape, Identity, If, Loop, Reshape, Scan, Shape, Size
  • Update to support float8e4m3fnuz, float8e5m2, float8e5m2fnuz, int4 and uint4:
    • Flatten, Pad, Squeeze, Transpose, Unsqueeze
  • Support blocked quantization. Support int4, uint4, int16, and uint16:
    • DequantizeLinear, QuantizeLinear,
  • Support bfloat16 and float16 scales. Support float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz quantized tensors:
    • QLinearMatMul,
  • Add stash_type attribute to GroupNormalization
• ai.onnx.ml Opset 5
  • Addeded new operator TreeEnsemble
  • Deprecated TreeEnsembleClassifier and TreeEnsembleRegressor
• IR Version 10
  • Enabled UINT4, INT4 support
• Support Python 3.12

In case a bug in ONNX is detected during integration of ONNX 1.16.0, please open a ONNX Bug Report and tag ONNX Release Manager @cjvolzka so that the bug is fixed in the ONNX release branch.

[hmc-cs-mdrissi]

Support bfloat16 and float16 scales. Support float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz quantized tensors:

I'll note that this fails with model using bfloat16 and latest versions of tf2onnx/onnx/onnxruntime. Error message looks like,

 File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tfonnx.py", line 459, in process_tf_graph
    main_g, subgraphs = graphs_from_tf(tf_graph, input_names, output_names, shape_override, const_node_values,
  File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tfonnx.py", line 474, in graphs_from_tf
    ordered_func = resolve_functions(tf_graph)
  File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tf_loader.py", line 784, in resolve_functions
    _, _, _, _, _, functions = tflist_to_onnx(tf_graph, {})
  File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tf_utils.py", line 443, in tflist_to_onnx
    onnx_tensor = tf_to_onnx_tensor(value, name=port_name(node.name))
  File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/tf2onnx/tf_utils.py", line 65, in tf_to_onnx_tensor
    return numpy_helper.from_array(np_data, name=name)
  File "/home/mdrissi/.venvs/bento/lib/python3.9/site-packages/onnx/numpy_helper.py", line 324, in from_array
    raise RuntimeError(
RuntimeError: Numpy data type not understood yet: bfloat16

Unsure if fix is better then in tf2onnx side in tf_to_onnx_tensor or in onnx side in numpy_helper. I see a comment

# NumPy doesn't have BFLOAT16.

in onnx and this issue is still open so I think fix makes more sense in tf2onnx and the assumption that tensorflow tensor can be converted to numpy and then to onnx is not true for bfloat16.