Add BatchNorm support

[zaleslaw]

System information

• 1.15 and 2.3:
• Are you willing to contribute it (Yes):

Describe the feature and the current behavior/state.
The problem is described in the mailing thread.

I have next problem: trying to repeat one of the modern CNN architectures on Java API. Most of them are using BatchNormalization as a popular layer with tf.nn.batchNormalization() op.

I trying to use old operands like BatchNormWithGlobalNormalization

I've used but got Exception in thread "main" org.tensorflow.exceptions.TFUnimplementedException: Op BatchNormWithGlobalNormalization is not available in GraphDef version 175. It has been removed in version 9. Use tf.nn.batch_normalization().
at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK(AbstractTF_Status.java:99)

This was deperecated years ago, but we have it in 1.15 and 2.x APIs.

The possible solutions:

1. To implement all required stuff in Java Ops
2. Add gradients and contributions in TF Core (gradients and another stuff) if it's possible

[aday00]

Really appreciate this BatchNorm effort @zaleslaw! Would it help if I made a separate ticket to ask for a Concat layer, because gradients are missing from Concat?

For now, I've made a "fake concat" using padding and a sum, like this, in Scala:

val padded1 = tf.withName("padded1").pad(some_input_of_512_dimensions,
                                         tf.constant(Array(Array(0,0), Array(1024,0))), tf.constant(0.0f)) 
val padded2 = tf.withName("padded2").pad(some_other_input_of_1024_dimensions,
                                         tf.constant(Array(Array(0,0), Array(0,512))), tf.constant(0.0f))
val fake_concat = tf.withName("fake_concat").math.add(padded1, padded2) // add these two vectors together, effectively concat'ing them.  tf.concat(...) doesn't itself have gradients implemented, so doesn't work during training.

Pseudocode for this "fake concat" is in our earlier discussion https://groups.google.com/a/tensorflow.org/g/jvm/c/TTuT3yzoKWs/m/pTQX1w_XAgAJ

[zaleslaw]

Agree about separate ticket, please make it вс, 22 нояб. 2020 г., 9:40 aday00 <notifications@github.com>:

…

Really appreciate this BatchNorm effort @zaleslaw <https://github.com/zaleslaw>! Would it help if I made a separate ticket to ask for a Concat layer, because gradients are missing from Concat? For now, I've made a "fake concat" using padding and a sum, like this, in Scala: val padded1 = tf.withName("padded1").pad(some_input_of_512_dimensions, tf.constant(Array(Array(0,0), Array(1024,0))), tf.constant(0.0f)) val padded2 = tf.withName("padded2").pad(some_other_input_of_1024_dimensions, tf.constant(Array(Array(0,0), Array(0,512))), tf.constant(0.0f)) val fake_concat = tf.withName("fake_concat").math.add(padded1, padded2) // add these two vectors together, effectively concat'ing them. tf.concat(...) doesn't itself have gradients implemented, so doesn't work during training. Pseudocode for this "fake concat" is in our earlier discussion https://groups.google.com/a/tensorflow.org/g/jvm/c/TTuT3yzoKWs/m/pTQX1w_XAgAJ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJEUHOT3I7L5KFYIK5GVG3SRCW43ANCNFSM4S4Z2RHA> .