Multipass changes used subnet across boots

[morphis]

Yesterday I've setup two machines with multipass (beta, rev 334) and deployed them via juju successfully. Also multipass is configured to use the libvirtd driver via $ snap set multipass driver=LIBVIRT

Now today (after a reboot of the host machine) both machines have IPs of a different subnet:

They were added to juju yesterday as:

0        started  10.16.239.128  manual:10.16.239.128  xenial      Manually provisioned machine
1        started  10.16.239.209  manual:10.16.239.209  xenial      Manually provisioned machine

but are now reported as

Name                    State       IPv4             Release
ams0                    RUNNING     10.150.176.128   Ubuntu 16.04 LTS
lxd0                    RUNNING     10.150.176.209   Ubuntu 16.04 LTS

The last part of the address is still the same but the subnet has changed. I've double checked that the addresses inside the VMs are the same as reported by multipass.

Interestingly the file multipass seems to use to store the used subnet hasn't changed since yesterday

$ ls -alh /var/snap/multipass/common/data/multipassd/vm-ips/multipass_subnet
-rw-r--r-- 1 root root 10 Aug 29 08:08 /var/snap/multipass/common/data/multipassd/vm-ips/multipass_subnet

and also reports a completely different subnet as the ones above

$ sudo cat /var/snap/multipass/common/data/multipassd/vm-ips/multipass_subnet 
10.202.213

I've uploaded logs to https://pastebin.canonical.com/p/yrhWxB6dgs/ You will see manual restarts of multipass in there as it failed to startup this morning.

[townsend2010]

Not really sure what happened, but it seems libvirt lost its brains somewhere along the way.

libvirt: Network Driver error : Network not found: no network with matching name 'default'

This is where multipassd probes libvirt to see if a default network is available, and libvirtd saying there is not. So multipassd then creates a new network. Which is probably where the new subnet comes in.

In summary, it looks like a default network was created with the 10.16.239 subnet, then somehow that default network was lost and then on the subsequent restart of multipassd, a new default network with the 10.150.176 was created. I've never seen the default network get deleted though, so I'm not really sure how to proceed on this as I've rebooted many, many times without issues like this.

Also, as an aside, multipass_subnet is a legacy file used when using the qemu backend.

[morphis]

@townsend2010 It happened again this morning. I've verified with virsh that there is no default network available on libvirt directly after the fresh boot. See https://pastebin.canonical.com/p/XGhj7zh3hR/ for more details.

[townsend2010]

@morphis,
I just reproduced this issue. Will dig further. Thanks!

[townsend2010]

Ok, I found the issue. This has been a day 1 issue, but I guess it didn't show up until now since I guess no one else cared about the subnet persisting:)
The particular libvirt api call I'm using doesn't persist the network across libvirtd restarts. I'm testing a fix now.

[morphis]

@townsend2010 Thanks, will give it a try once in edge

[jovylle]

I am listening to this.