illumos subnet router doesn't route ICMP pings

[nshalman]

What is the issue?

My tailnet has two subnet routers, one that runs on a raspberry pi running linux, the other in an illumos zone.
If I disable the illumos one and enable the pi, my phone can ping an IP in the LAN subnet from outside the house.
If I disable the pi and enable the illumos one, my phone cannot ping that IP on the LAN (it can still do e.g. TCP just fine)

Also reported by a user on IRC.

Steps to reproduce

Create an illumos subnet router advertising a subnet
try to ping an IP on the other side of the subnet router

Are there any recent changes that introduced the issue?

No response

OS

Other

OS version

illumos

Tailscale version

1.80.2-sunos

Other software

No response

Bug report

No response

[extrowerk]

Ping.

[extrowerk]

I have installed my NAs in december, and I am sure I had ping working, so I have tested all release back to that date, but none works.
Any other idea?

[extrowerk]

Tested with v1.80.3-sunos, still not working.

[extrowerk]

Tested with v1.82.0, still not ok.

[extrowerk]

No change in 1.82.5

[extrowerk]

Maybe related: tailscale#16158

[extrowerk]

Stil lexists in v1.86.2

[extrowerk]

I have investigated a bit further:

I have advertised the routes in the OmniOS tailscale zone like:

/opt/tailscale/sbin/tailscale up --advertise-routes=10.0.0.0/24 --advertise-exit-node

With this ping doesn't works, so I have tried to relax it a bit like it was suggested here tailscale#16158 (comment):

/opt/tailscale/sbin/tailscale up --advertise-routes=10.0.0.0/23 --advertise-exit-node

Notice 23 instead of 24.

Connection still works, but no difference in pinging.

However looking at the debug logs I see the following:

/opt/tailscale/sbin/tailscale debug daemon-logs --verbose 1

(Pingin the Tailscale Zone IP from my Android phone):

(...)
Accept: ICMPv4{100.108.172.100:0 > 10.0.0.20:0} 84 icmp ok
[RATELIMIT] format("exec ping of %v failed in %v: %v") (2 dropped)
exec ping of 10.0.0.20 failed in 5.222878ms: exit status 1
Accept: ICMPv4{100.108.172.100:0 > 10.0.0.20:0} 84 icmp ok
exec ping of 10.0.0.20 failed in 4.948089ms: exit status 1
exec ping of 10.0.0.20 failed in 4.86823ms: exit status 1
exec ping of 10.0.0.20 failed in 4.101283ms: exit status 1
(...)

(Pingin the OmniOS main IP from my Android phone):
(...)
Accept: ICMPv4{100.108.172.100:0 > 10.0.0.10:0} 84 icmp ok
[RATELIMIT] format("exec ping of %v failed in %v: %v") (1 dropped)
exec ping of 10.0.0.10 failed in 5.003351ms: exit status 1
Accept: ICMPv4{100.108.172.100:0 > 10.0.0.10:0} 84 icmp ok
exec ping of 10.0.0.10 failed in 4.906238ms: exit status 1
exec ping of 10.0.0.10 failed in 5.010774ms: exit status 1
exec ping of 10.0.0.10 failed in 4.695172ms: exit status 1
(...)

So ping arrives, but sending the pongs fails somewhere.