make sure we have enough "services" pool IPs for Nexus upgrades

[davepacheco]

This is a simpler case of #8511 and more important for the initial delivery of self-service upgrade. Unlike the other components, Nexus has to be deployed by adding new ones and removing old ones. Technically, only one set of instances (the old ones or the new ones) needs external IPs at once.

Either we need to:

• dynamically attach/detach external IPs to Nexus zones (ideal, because then the IPs wouldn't change across an upgrade). I don't know how hard this would be.
• have enough IPs in the external services pool for 2x the normal number of Nexus zones. This involves:
  • verifying that we already do for existing deployments
  • making sure we validate that at RSS time

Marking this for R16 because I want to make sure that we check existing deployments during the R16 upgrade.

[askfongjojo]

@davepacheco - I can add the checks to the v16 rack update runbook. Would these database queries suffice?

select first_address, last_address from ip_pool_range
join ip_pool on ip_pool.id = ip_pool_range.ip_pool_id
where ip_pool.name = 'oxide-service-pool';

which returns something like

  first_address | last_address
----------------+---------------
  172.20.30.1   | 172.20.30.10
(1 row)

and

select kind, ip from external_ip
join ip_pool on ip_pool.id = external_ip.ip_pool_id
where ip_pool.name = 'oxide-service-pool'
and external_ip.time_deleted is NULL
order by ip;

which gives you

    kind   |     ip
-----------+--------------
  floating | 172.20.30.1
  floating | 172.20.30.2
  floating | 172.20.30.3
  floating | 172.20.30.4
  snat     | 172.20.30.5
  snat     | 172.20.30.5
(6 rows)

[davepacheco]

Thank you! That looks about right but I'm not that familiar with these tables so it'd be nice to get a +1 from somebody else.

[jgallagher]

@davepacheco - I can add the checks to the v16 rack update runbook. Would these database queries suffice?

select first_address, last_address from ip_pool_range
join ip_pool on ip_pool.id = ip_pool_range.ip_pool_id
where ip_pool.name = 'oxide-service-pool';

which returns something like

  first_address | last_address
----------------+---------------
  172.20.30.1   | 172.20.30.10
(1 row)

This one looks right to me. I believe this gets populated during the RSS handoff and never touched again.

select kind, ip from external_ip
join ip_pool on ip_pool.id = external_ip.ip_pool_id
where ip_pool.name = 'oxide-service-pool'
and external_ip.time_deleted is NULL
order by ip;

which gives you

    kind   |     ip
-----------+--------------
  floating | 172.20.30.1
  floating | 172.20.30.2
  floating | 172.20.30.3
  floating | 172.20.30.4
  snat     | 172.20.30.5
  snat     | 172.20.30.5
(6 rows)

I think this is right? This info is also available via omdb; e.g., on dogfood:

root@oxz_switch0:~# omdb db network list-eips  | grep service
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (155.0.0)
instance with id 2e79d198-51b9-4190-b112-a581b97f424b not found
 172.20.26.1/32     0/65535      floating   Attached  service     edd99650-5df1-4241-815d-253e4ef2399c  ExternalDns                                  in service
 172.20.26.2/32     0/65535      floating   Attached  service     f500d564-c40a-4eca-ac8a-a26b435f2037  ExternalDns                                  in service
 172.20.26.3/32     0/65535      floating   Attached  service     65a11c18-7f59-41ac-b9e7-680627f996e7  Nexus                                        in service
 172.20.26.4/32     0/65535      floating   Attached  service     20b100d0-84c3-4119-aa9b-0c632b0b6a3a  Nexus                                        in service
 172.20.26.5/32     0/65535      floating   Attached  service     2898657e-4141-4c05-851b-147bffc6bbbd  Nexus                                        in service
 172.20.26.6/32     0/16383      SNAT       Attached  service     c3ec3d1a-3172-4d36-bfd3-f54a04d5ba55  Ntp                                          in service
 172.20.26.7/32     16384/32767  SNAT       Attached  service     6ea2684c-115e-48a6-8453-ab52d1cecd73  Ntp                                          in service

[askfongjojo]

Thank you both for the feedback. I opted for the direct SQL query for the second check as I'm not 100% sure what will be a "safe" fetch-limit to use (and we are in CRDB CLI anyway).

The output from the checks will be recorded in this internal ticket.