[Apple M1] Instance no longer get an IP address when it previously worked

[townsend2010]

This is separating out the ongoing issues seen in #1857 where for some reason, instances are no longer getting an IP address when it previously worked.

[townsend2010]

Hi @adamcstephens & @lethargosapatheia,

I've moved the thread to this new issue. One thing I'd like for you to check is when there is a Multipass instance launched, do you see a bridge100 in the output of ifconfig? If so, could you post the output of ifconfig bridge100?

Thanks!

[lethargosapatheia]

Here it is:

$ multipass launch --name testing
Starting testing /

bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 3e:06:30:20:54:64
	inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
	inet6 fe80::8ae:9486:dbeb:1196%bridge100 prefixlen 64 optimistic secured scopeid 0x14
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x0
	member: en7 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 19 priority 0 path cost 0
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active

[adamcstephens]

I ran a tcpdump on that interface and the VM is requesting an IPv4 address:

─❯ sudo tcpdump -i bridge100 port bootpc or port bootps
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge100, link-type EN10MB (Ethernet), capture size 262144 bytes
21:04:24.858843 IP 0.0.0.0.bootpc > broadcasthost.bootps: BOOTP/DHCP, Request from 52:54:00:b1:40:8f (oui Unknown), length 300
21:05:29.811132 IP 0.0.0.0.bootpc > broadcasthost.bootps: BOOTP/DHCP, Request from 52:54:00:b1:40:8f (oui Unknown), length 300
21:06:33.347818 IP 0.0.0.0.bootpc > broadcasthost.bootps: BOOTP/DHCP, Request from 52:54:00:b1:40:8f (oui Unknown), length 300
21:07:37.779799 IP 0.0.0.0.bootpc > broadcasthost.bootps: BOOTP/DHCP, Request from 52:54:00:b1:40:8f (oui Unknown), length 300

Also IPv6 seems to work, I assume this is the VM based off the tcpdump data:

─❯ ping6 fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f
PING6(56=40+8+8 bytes) fd33:5c04:3ba9:a6e2:1c8c:69bb:f02a:b1c7 --> fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f
16 bytes from fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f, icmp_seq=0 hlim=64 time=1.531 ms
16 bytes from fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f, icmp_seq=1 hlim=64 time=0.468 ms
16 bytes from fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f, icmp_seq=2 hlim=64 time=0.918 ms
^C
--- fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.468/0.972/1.531/0.436 ms

~ took 2s 
─❯ ssh fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f           
The authenticity of host 'fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f (fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f)' can't be established.
ECDSA key fingerprint is SHA256:YZ3EfOyYhtG3g+8vRKmwDb/kf3caweRTnGRcYhQTrQg.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f' (ECDSA) to the list of known hosts.
adam@fd33:5c04:3ba9:a6e2:5054:ff:feb1:408f: Permission denied (publickey).

Is InternetSharing providing DHCP for the VMs? Systems Preferences gives me the indication it's not fully configured:

I also pulled this out of my system.log in #1857. Could this be a relevant error?

Sep 27 15:34:16 mycomputer InternetSharing[11079]: DEPRECATED USE in libdispatch client: Setting timer interval to 0 requests a 1ns timer, did you mean FOREVER (a one-shot timer)?; set a breakpoint on _dispatch_bug_deprecated to debug

Is there somewhere I can learn more about how InternetSharing is configured? Does it need to be notified of the interface somehow?

[Saviq]

I ran a tcpdump on that interface and the VM is requesting an IPv4 address:

Also IPv6 seems to work, I assume this is the VM based off the tcpdump data:

All that shows that the network is set up correctly from our side. Thanks a lot for the dig!

Is InternetSharing providing DHCP for the VMs? Systems Preferences gives me the indication it's not fully configured:

Yes, but not in a way you can influence. If you enable Internet Sharing here it will actually prevent instances from starting altogether (the network interface will not start).

How it's set up is a bit of a black box - just starting the interface using the vmnet API is what triggers the InternetSharing process that should be serving DHCP. That's why we asked about vmcli and vftool, as they use the same networking API.

I also pulled this out of my system.log in #1857. Could this be a relevant error?

Sep 27 15:34:16 mycomputer InternetSharing[11079]: DEPRECATED USE in libdispatch client: Setting timer interval to 0 requests a 1ns timer, did you mean FOREVER (a one-shot timer)?; set a breakpoint on _dispatch_bug_deprecated to debug

Don't think so, no.

Is there somewhere I can learn more about how InternetSharing is configured? Does it need to be notified of the interface somehow?

And again, that's what QEMU is doing, but macOS seems to not be doing the right thing :/

[Saviq]

FWIW, the config of the DHCP lies in /etc/bootpd.list (even though bootpd itself isn't in play in Big Sur any more), my contents are:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Subnets</key>
	<array>
		<dict>
			<key>_creator</key>
			<string>com.apple.NetworkSharing</string>
			<key>allocate</key>
			<true/>
			<key>dhcp_domain_name_server</key>
			<array>
				<string>192.168.65.1</string>
			</array>
			<key>dhcp_router</key>
			<string>192.168.65.1</string>
			<key>interface</key>
			<string>bridge100</string>
			<key>lease_max</key>
			<integer>86400</integer>
			<key>lease_min</key>
			<integer>86400</integer>
			<key>name</key>
			<string>192.168.65/24</string>
			<key>net_address</key>
			<string>192.168.65.0</string>
			<key>net_mask</key>
			<string>255.255.255.0</string>
			<key>net_range</key>
			<array>
				<string>192.168.65.2</string>
				<string>192.168.65.254</string>
			</array>
		</dict>
	</array>
	<key>bootp_enabled</key>
	<false/>
	<key>detect_other_dhcp_server</key>
	<array>
		<string>bridge100</string>
	</array>
	<key>dhcp_enabled</key>
	<array>
		<string>bridge100</string>
	</array>
	<key>dhcp_ignore_client_identifier</key>
	<true/>
	<key>ignore_allow_deny</key>
	<array>
		<string>bridge100</string>
	</array>
	<key>use_server_config_for_dhcp_options</key>
	<true/>
</dict>
</plist>

The file gets recreated if moved away, but maybe it's corrupted in your case somehow? Try killing InternetSharing and moving the file away, and see if Multipass instances work again?

[Saviq]

Hi @adamcstephens & @lethargosapatheia, did you have a chance to review what I wrote above? Maybe tried moving the file away to test?

[adamcstephens]

@Saviq yes I moved the file and it was recreated with the same contents. It looks similar to yours but with a different subnet.

[Saviq]

@adamcstephens …and it didn't help?

[townsend2010]

Also @adamcstephens, what is the subnet listed in your file and does it match the vmnet info that is printed out in the multipassd log? The log info will look similar to:

[2021-09-27T15:13:14.340] [warning] [amazing-jerboa] qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: Started vmnet interface with configuration:
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: MTU:              1500
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: Max packet size:  1514
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: MAC:              ca:5e:03:51:a5:84
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: DHCP IPv4 start:  192.168.64.1
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: DHCP IPv4 end:    192.168.64.254
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: IPv4 subnet mask: 255.255.255.0
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:44:a0:8d: info: UUID:             B7B80AF9-7FCC-4EFB-BBCE-C0B734029EA9

Thanks!

[adamcstephens]

@Saviq correct, it did not help.

@townsend2010 yes, it does match

─❯ tail -n 20 /Library/Logs/Multipass/multipassd.log
[2021-09-30T12:20:56.149] [info] [magical-phoenix] process state changed to Running
[2021-09-30T12:20:56.149] [debug] [qemu-system-aarch64] [96084] started: qemu-system-aarch64 -machine virt,highmem=off -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7 -cpu cortex-a72 -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/magical-phoenix/ubuntu-20.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/magical-phoenix/cloud-init-config.iso
[2021-09-30T12:20:56.149] [info] [magical-phoenix] process started
[2021-09-30T12:20:56.150] [debug] [magical-phoenix] Waiting for SSH to be up
[2021-09-30T12:20:56.720] [warning] [magical-phoenix] qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: Started vmnet interface with configuration:
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: MTU:              1500
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: Max packet size:  1514
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: MAC:              da:ae:20:83:00:11
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: DHCP IPv4 start:  192.168.64.1
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: DHCP IPv4 end:    192.168.64.254
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: IPv4 subnet mask: 255.255.255.0
qemu-system-aarch64: -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:21:39:c7: info: UUID:             47DDB02F-3260-4F9C-8DF4-A5A1E70A7DA5

[2021-09-30T12:20:56.720] [warning] [qemu-system-aarch64] 
[2021-09-30T12:20:56.721] [debug] [magical-phoenix] QMP: {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 6}, "package": ""}, "capabilities": ["oob"]}}

[2021-09-30T12:20:56.758] [debug] [magical-phoenix] QMP: {"return": {}}

[2021-09-30T12:21:10.492] [debug] [magical-phoenix] QMP: {"timestamp": {"seconds": 1633018870, "microseconds": 492559}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[6]/virtio-backend"}}

─❯ cat /etc/bootpd.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Subnets</key>
	<array>
		<dict>
			<key>_creator</key>
			<string>com.apple.NetworkSharing</string>
			<key>allocate</key>
			<true/>
			<key>dhcp_domain_name_server</key>
			<array>
				<string>192.168.64.1</string>
			</array>
			<key>dhcp_router</key>
			<string>192.168.64.1</string>
			<key>interface</key>
			<string>bridge100</string>
			<key>lease_max</key>
			<integer>86400</integer>
			<key>lease_min</key>
			<integer>86400</integer>
			<key>name</key>
			<string>192.168.64/24</string>
			<key>net_address</key>
			<string>192.168.64.0</string>
			<key>net_mask</key>
			<string>255.255.255.0</string>
			<key>net_range</key>
			<array>
				<string>192.168.64.2</string>
				<string>192.168.64.254</string>
			</array>
		</dict>
	</array>
	<key>bootp_enabled</key>
	<false/>
	<key>detect_other_dhcp_server</key>
	<array>
		<string>bridge100</string>
	</array>
	<key>dhcp_enabled</key>
	<array>
		<string>bridge100</string>
	</array>
	<key>dhcp_ignore_client_identifier</key>
	<true/>
	<key>ignore_allow_deny</key>
	<array>
		<string>bridge100</string>
	</array>
	<key>use_server_config_for_dhcp_options</key>
	<true/>
</dict>
</plist>

[adamcstephens]

vmcli exhibits the same behavior of no DHCP responses. I couldn't get vftool working to see its results.

[adamcstephens]

I am currently not experiencing this issue. It's possible a reboot solved this but I am not sure.

I have launched and destroyed multiple VMs over the past couple days without problem.

[Saviq]

Thanks @adamcstephens, that's great news.

@lethargosapatheia are you still seeing this problem? Had you tried to reboot yet?