trying bgp peer set with peer address already in use but different asn results in existing bgp peer disappearing #6472
Description
I am running a4x2 with:
- omicron
1059fe109fadbccf559e4adad134eaffcb67b85a - Oxide CLI version compiled from main with this commit:
$ oxide version
Oxide CLI 0.6.1+20240710.0
Built from commit: 786cd71463cc74b6e00faf3575095c0c5f5036aa
Oxide API: 20240821.0
Initial state:
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.30.1 65547 64502 Established 1day 1h 14m 40s 883ms
169.254.10.1 65547 64500 Established 1day 1h 14m 39s 826ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.40.1 65547 64502 Established 1day 41m 25s 496ms
169.254.20.1 65547 64500 Established 1day 59m 17s 875ms
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
Destination Nexthop Vlan Preference
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
169.254.30.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
I created a fake peer for address not yet used as a peer address with the ASN already in use by the existing peers.
$ oxide system networking bgp peer set --rack 619c28eb-c688-4823-b846-a6f9ed25be12 --switch switch1 --port qsfp0 --addr 169.254.60.1 --bgp-config as65547
Resulting state after above command (I see 169.254.60.1 has appeared in the output):
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.30.1 65547 64502 Established 1day 1h 18m 51s 221ms
169.254.10.1 65547 64500 Established 1day 1h 18m 50s 165ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.20.1 65547 64500 Established 1day 1h 3m 28s 194ms
169.254.40.1 65547 64502 Established 1day 45m 35s 814ms
169.254.60.1 65547 0 Connect 6s 699ms
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.60.1 as65547 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
169.254.30.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
Now I try to create/set (another/same) fake peer to use an ASN not already in use by the existing peers.
$ oxide system networking bgp peer set --rack 619c28eb-c688-4823-b846-a6f9ed25be12 --switch switch1 --port qsfp0 --addr 169.254.60.1 --bgp-config as65548
Error Response: status: 409 Conflict; headers: {"content-type": "application/json", "x-request-id": "f559d782-5ae7-4055-9ada-e0375f13c8be", "content-length": "155", "date": "Thu, 29 Aug 2024 01:37:43 GMT"}; value: Error { error_code: Some("Conflict"), message: "a different asn is already configured on this switch", request_id: "f559d782-5ae7-4055-9ada-e0375f13c8be" }
Resulting state after above command (I see 169.254.60.1 has disappeared from the output of the first command, but as65548 has appeared in the output of the second command):
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.30.1 65547 64502 Established 1day 1h 21m 27s 616ms
169.254.10.1 65547 64500 Established 1day 1h 21m 26s 559ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.20.1 65547 64500 Established 1day 1h 6m 4s 590ms
169.254.40.1 65547 64502 Established 1day 48m 12s 210ms
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.60.1 as65548 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
169.254.30.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
I performed the same set of operations on my real peer 169.254.40.1, and it had the same behavior.
Initial state:
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.30.1 65547 64502 Established 1day 1h 41m 53s 255ms
169.254.10.1 65547 64500 Established 1day 1h 41m 52s 198ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.40.1 65547 64502 Established 1day 1h 8m 37s 881ms
169.254.20.1 65547 64500 Established 1day 1h 26m 30s 261ms
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.60.1 as65548 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
169.254.30.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
Try to create/set with same address as existing real peer, different ASN.
$ oxide system networking bgp peer set --rack 619c28eb-c688-4823-b846-a6f9ed25be12 --switch switch1 --port qsfp0 --addr 169.254.40.1 --bgp-config as65548
Error Response: status: 409 Conflict; headers: {"content-type": "application/json", "x-request-id": "30ee7ed5-b4c0-4252-a732-f29c7207ca5e", "content-length": "155", "date": "Thu, 29 Aug 2024 01:58:57 GMT"}; value: Error { error_code: Some("Conflict"), message: "a different asn is already configured on this switch", request_id: "30ee7ed5-b4c0-4252-a732-f29c7207ca5e" }
Resulting state after above command:
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.10.1 65547 64500 Established 1day 1h 42m 26s 21ms
169.254.30.1 65547 64502 Established 1day 1h 42m 27s 78ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.20.1 65547 64500 Established 1day 1h 27m 4s 89ms
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65548 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
169.254.60.1 as65548 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
169.254.30.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
Try to create/set with same address as existing real peer, same real ASN that was previously in use.
$ oxide system networking bgp peer set --rack 619c28eb-c688-4823-b846-a6f9ed25be12 --switch switch1 --port qsfp0 --addr 169.254.40.1 --bgp-config as65547
Error Response: status: 409 Conflict; headers: {"content-type": "application/json", "x-request-id": "f0922504-8d98-4271-97e9-9523fe3da70c", "content-length": "155", "date": "Thu, 29 Aug 2024 02:19:52 GMT"}; value: Error { error_code: Some("Conflict"), message: "a different asn is already configured on this switch", request_id: "f0922504-8d98-4271-97e9-9523fe3da70c" }
Resulting state (169.254.40.1 now using 65547 and as65547):
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.30.1 65547 64502 Established 1day 2h 23m 37s 638ms
169.254.10.1 65547 64500 Established 1day 2h 23m 36s 581ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.40.1 65547 64502 Established 20m 11s 699ms
169.254.20.1 65547 64500 Established 1day 2h 8m 14s 710ms
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65547 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
169.254.60.1 as65548 [no filtering] [no filtering] [] 0 0 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
169.254.30.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
Destination Nexthop Vlan Preference
For reference, here is the ASN information:
$ oxide system networking bgp config list
[
{
"asn": 65551,
"description": "hello5",
"id": "1c013259-97ff-426f-8414-7cacf08dcae4",
"name": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz12345678901",
"time_created": "2024-08-29T00:07:02.774536Z",
"time_modified": "2024-08-29T00:07:02.774536Z"
}, {
"asn": 65547,
"description": "BGP config for AS 65547",
"id": "a3b1a17e-7167-47a2-9dff-5ce8cb5ab38b",
"name": "as65547",
"time_created": "2024-08-28T00:16:10.327972Z",
"time_modified": "2024-08-28T00:16:10.327972Z"
}, {
"asn": 65548,
"description": "hello",
"id": "5b278ef8-7da2-4c84-859e-e955bbb202ed",
"name": "as65548",
"time_created": "2024-08-28T23:28:31.655895Z",
"time_modified": "2024-08-28T23:28:31.655895Z"
}, {
"asn": 65549,
"description": "hello3",
"id": "8773e937-dea8-422f-8dcf-5add050081eb",
"name": "as65549",
"time_created": "2024-08-28T23:37:09.307159Z",
"time_modified": "2024-08-28T23:37:09.307159Z"
}, {
"asn": 65550,
"description": "0",
"id": "fef48d7f-97a3-4553-aade-42c1dd2386bb",
"name": "as65550",
"time_created": "2024-08-28T23:57:34.067397Z",
"time_modified": "2024-08-28T23:57:34.067397Z"
}, {
"asn": 4294967295,
"description": "hello6",
"id": "26a80a18-a43c-4228-a62c-e4e6a9a54bb0",
"name": "as65552",
"time_created": "2024-08-29T00:14:51.310793Z",
"time_modified": "2024-08-29T00:14:51.310793Z"
}
]
Resulting state:
On g3 (switch1):
root@oxz_switch:~# mgadm bgp c r l
[
Router {
asn: 65547,
graceful_shutdown: false,
id: 65547,
listen: "[::]:179",
},
]
And for reference, on g0 (switch0):
root@oxz_switch:~# mgadm bgp c r l
[
Router {
asn: 65547,
graceful_shutdown: false,
id: 65547,
listen: "[::]:179",
},
]