Caught an unhandled exception: unable to create directory '/data/cache'

[wolfenstein87]

Describe the bug
Unable to start multipass after following the instructions in #1789

Expected behavior
Multipass to start with the new storage directory location configured

Logs

Aug 17 19:58:03 woof-lab systemd[1]: Started Service for snap application multipass.multipassd.
Aug 17 19:58:03 woof-lab multipassd[17309]: Caught an unhandled exception: unable to create directory '/data/cache'
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 1.
Aug 17 19:58:03 woof-lab systemd[1]: Stopped Service for snap application multipass.multipassd.
Aug 17 19:58:03 woof-lab systemd[1]: Started Service for snap application multipass.multipassd.
Aug 17 19:58:03 woof-lab multipassd[17349]: Caught an unhandled exception: unable to create directory '/data/cache'
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 2.
Aug 17 19:58:03 woof-lab systemd[1]: Stopped Service for snap application multipass.multipassd.
Aug 17 19:58:03 woof-lab systemd[1]: Started Service for snap application multipass.multipassd.
Aug 17 19:58:03 woof-lab multipassd[17385]: Caught an unhandled exception: unable to create directory '/data/cache'
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 17 19:58:03 woof-lab systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 3.
Aug 17 19:58:03 woof-lab systemd[1]: Stopped Service for snap application multipass.multipassd.
Aug 17 19:58:03 woof-lab systemd[1]: Started Service for snap application multipass.multipassd.
Aug 17 19:58:04 woof-lab multipassd[17422]: Caught an unhandled exception: unable to create directory '/data/cache'
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 4.
Aug 17 19:58:04 woof-lab systemd[1]: Stopped Service for snap application multipass.multipassd.
Aug 17 19:58:04 woof-lab systemd[1]: Started Service for snap application multipass.multipassd.
Aug 17 19:58:04 woof-lab multipassd[17459]: Caught an unhandled exception: unable to create directory '/data/cache'
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 5.
Aug 17 19:58:04 woof-lab systemd[1]: Stopped Service for snap application multipass.multipassd.
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Start request repeated too quickly.
Aug 17 19:58:04 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 17 19:58:04 woof-lab systemd[1]: Failed to start Service for snap application multipass.multipassd.

Additional info

• OS: Ubuntu 22.04.1 LTS
• multipass 1.11.0-dev.74+g42aeb3e9

Additional context

root@server:/data# ls -alh
total 36K
drwxrwxrwx  6 root multipass 4.0K Aug 17 19:45 .
drwxr-xr-x 20 root root      4.0K Aug  9 14:35 ..
drwxr-xr-x  2 root root      4.0K Aug 17 19:45 cache
drwxr-xr-x  2 root root      4.0K Aug 17 16:32 conf-sharedhome
drwxr-xr-x  2 root root      4.0K Aug 17 16:32 jira-sharedhome
drwx------  2 root multipass  16K Aug  8 16:00 lost+found

[townsend2010]

Hi @wolfenstein87,

I think you need to chown root.root for /data as well like done in #1789 (comment). Hopefully that will help. Snap confinement is very strict 😉

[wolfenstein87]

Hi @townsend2010,

Yea, I had done that while following another "how-to" article that also referenced #1789. However, this did not allow mutlipass to start, and it's produced the same errors.

Here is what I have in override.conf

[Service]
Environment=MULTIPASS_STORAGE=/data/

root@server:/# ls -alh /data/
total 40K
drwxrwxrwx  7 root root 4.0K Aug 17 20:19 .
drwxr-xr-x 20 root root 4.0K Aug  9 14:35 ..
drwxr-xr-x  2 root root 4.0K Aug 17 19:45 cache
drwxr-xr-x  2 root root 4.0K Aug 17 16:32 conf-sharedhome
drwxr-xr-x  2 root root 4.0K Aug 17 16:32 jira-sharedhome
drwx------  2 root root  16K Aug  8 16:00 lost+found
drwxr-xr-x  2 root root 4.0K Aug 17 20:19 postgres

root@server:/# df -h
Filesystem                    Size  Used Avail Use% Mounted on
tmpfs                         4.8G  1.9M  4.8G   1% /run
/dev/mapper/ubuntu--vg-lv--0  227G   14G  201G   7% /
tmpfs                          24G     0   24G   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
/dev/sda2                     2.0G  262M  1.6G  15% /boot
/dev/sdb                      3.2T   44K  3.0T   1% /data
tmpfs                         4.8G  4.0K  4.8G   1% /run/user/0
tmpfs                          24G     0   24G   0% /run/qemu

[townsend2010]

Hmm, ok, I really think snap confinement is tripping this up and won't allow access there. That said, we do connect to the removable-media interface and so it may be possible to create a symlink in one of those directories to your /data directory and point MULTIPASS_STORAGE to that symlink. I haven't tried it, but hopefully it will work 🤞

[wolfenstein87]

Yea, it doesn't seem to like that either. I guess I could change the mount point of that volume to be in /mnt. This is a home lab machine, so it's not holding any crucial data to be screwed up.

Aug 18 13:40:33 woof-lab multipassd[3013]: Caught an unhandled exception: unable to create directory '/mnt/multipass/cache'
Aug 18 13:40:33 woof-lab systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Aug 18 13:40:33 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 18 13:40:33 woof-lab systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 5.
Aug 18 13:40:33 woof-lab systemd[1]: Stopped Service for snap application multipass.multipassd.
Aug 18 13:40:33 woof-lab systemd[1]: snap.multipass.multipassd.service: Start request repeated too quickly.
Aug 18 13:40:33 woof-lab systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Aug 18 13:40:33 woof-lab systemd[1]: Failed to start Service for snap application multipass.multipassd.
root@server:/mnt# mkdir multipass/cache
root@server:/mnt# ls -l multipass
lrwxrwxrwx 1 root root 15 Aug 18 13:38 multipass -> /data/multipass
root@server:/mnt# cd multipass
root@server:/mnt/multipass# ls -alh
total 12K
drwxr-xr-x 3 root root 4.0K Aug 18 13:47 .
drwxrwxrwx 8 root root 4.0K Aug 18 13:37 ..
drwxr-xr-x 2 root root 4.0K Aug 18 13:47 cache
root@server:/mnt/multipass# ls -l /data/multipass/
total 4
drwxr-xr-x 2 root root 4096 Aug 18 13:47 cache

[townsend2010]

Ok, darn. I think it may be due to /data/ being on another physical disk and Apparmor denying that. That's too bad....

[wolfenstein87]

I'm not very familiar with Apparmor. Is there a way to modify Apparmor's configuration to allow it?

[townsend2010]

Is there a way to modify Apparmor's configuration to allow it?

Unfortunately, not very easily and would take changes to Apparmor profiles under snadp control. That's really not something we should get in to.

[wolfenstein87]

That was it. The new storage directory would need to live in /home or /mnt in order for it to work. This is definitely an interesting limitation that I wouldn't expect from a VM application.

root@server:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
tmpfs                         4.8G  1.9M  4.8G   1% /run
/dev/mapper/ubuntu--vg-lv--0  227G   15G  201G   7% /
tmpfs                          24G   28K   24G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                          24G     0   24G   0% /run/qemu
/dev/sda2                     2.0G  262M  1.6G  15% /boot
/dev/sdb                      3.2T   67M  3.0T   1% /mnt/data
tmpfs                         4.8G  4.0K  4.8G   1% /run/user/0


Aug 18 13:59:11 woof-lab systemd[1]: Started Service for snap application multipass.multipassd.
Aug 18 13:59:13 woof-lab multipassd[1113]: Unable to determine subnet for the mpqemubr0 subnet
Aug 18 13:59:13 woof-lab multipassd[1113]: Using AppArmor support
Aug 18 13:59:14 woof-lab multipassd[1113]: Starting dnsmasq
Aug 18 13:59:14 woof-lab multipassd[1113]: Applied AppArmor policy: multipass.dnsmasq
Aug 18 13:59:14 woof-lab multipassd[1113]: [1681] started: dnsmasq --keep-in-foreground --strict-order --bind-interfaces --pid-file --domain=multipass ->
Aug 18 13:59:14 woof-lab dnsmasq[1681]: started, version 2.80 cachesize 150
Aug 18 13:59:14 woof-lab dnsmasq[1681]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth nettlehash DNSS>
Aug 18 13:59:14 woof-lab dnsmasq-dhcp[1681]: DHCP, IP range 10.112.234.2 -- 10.112.234.254, lease time infinite
Aug 18 13:59:14 woof-lab dnsmasq-dhcp[1681]: DHCP, sockets bound exclusively to interface mpqemubr0
Aug 18 13:59:14 woof-lab dnsmasq[1681]: using local addresses only for domain multipass
Aug 18 13:59:14 woof-lab dnsmasq[1681]: reading /etc/resolv.conf
Aug 18 13:59:14 woof-lab dnsmasq[1681]: using local addresses only for domain multipass
Aug 18 13:59:14 woof-lab dnsmasq[1681]: using nameserver 127.0.0.53#53
Aug 18 13:59:14 woof-lab dnsmasq[1681]: read /etc/hosts - 7 addresses
Aug 18 13:59:14 woof-lab dnsmasq-dhcp[1681]: read /mnt/data/multipass/data/network/dnsmasq.hosts
Aug 18 13:59:14 woof-lab multipassd[1113]: [1685] started: iptables-nft --wait -t filter --list-rules
Aug 18 13:59:14 woof-lab multipassd[1113]: Using iptables-nft for firewall rules.
Aug 18 13:59:14 woof-lab multipassd[1113]: [1686] started: iptables-nft --wait -t filter --list-rules
Aug 18 13:59:14 woof-lab multipassd[1113]: [1687] started: iptables-nft --wait -t nat --list-rules
Aug 18 13:59:14 woof-lab multipassd[1113]: [1688] started: iptables-nft --wait -t mangle --list-rules
Aug 18 13:59:14 woof-lab multipassd[1113]: [1689] started: iptables-nft --wait -t raw --list-rules
Aug 18 13:59:14 woof-lab multipassd[1113]: [1690] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol udp --dport >
Aug 18 13:59:14 woof-lab multipassd[1113]: [1692] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol udp --dport >
Aug 18 13:59:14 woof-lab multipassd[1113]: [1693] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol tcp --dport >
Aug 18 13:59:14 woof-lab multipassd[1113]: [1694] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol udp --spor>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1695] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol udp --spor>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1696] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol tcp --spor>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1697] started: iptables-nft --wait -t mangle --insert POSTROUTING --out-interface mpqemubr0 --protocol udp ->
Aug 18 13:59:14 woof-lab multipassd[1113]: [1698] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.112.234.0/24 --destination 224.0.>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1699] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.112.234.0/24 --destination 255.25>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1700] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.112.234.0/24 ! --destination 10.1>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1701] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.112.234.0/24 ! --destination 10.1>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1702] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.112.234.0/24 ! --destination 10.1>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1703] started: iptables-nft --wait -t filter --insert FORWARD --destination 10.112.234.0/24 --out-interface >
Aug 18 13:59:14 woof-lab multipassd[1113]: [1704] started: iptables-nft --wait -t filter --insert FORWARD --source 10.112.234.0/24 --in-interface mpqemu>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1705] started: iptables-nft --wait -t filter --insert FORWARD --in-interface mpqemubr0 --out-interface mpqem>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1706] started: iptables-nft --wait -t filter --append FORWARD --in-interface mpqemubr0 --jump REJECT --rejec>
Aug 18 13:59:14 woof-lab multipassd[1113]: [1707] started: iptables-nft --wait -t filter --append FORWARD --out-interface mpqemubr0 --jump REJECT --reje>
Aug 18 13:59:21 woof-lab multipassd[1113]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket
Aug 18 13:59:21 woof-lab multipassd[1113]: QIODevice::write (QFile, "/mnt/data/multipass/cache/vault/multipassd-image-records.json"): device not open
Aug 18 13:59:21 woof-lab multipassd[1113]: Starting Multipass 1.11.0-dev.76+g5069a6e9
Aug 18 13:59:21 woof-lab multipassd[1113]: Daemon arguments: /snap/multipass/7762/bin/multipassd --verbosity debug --logger platform

[townsend2010]

Hey @wolfenstein87,

Glad that is solved!

This is definitely an interesting limitation that I wouldn't expect from a VM application.

This is not a limitation of Multipass per se, but is a limitation of Snaps that we have to live with unfortunately. We'd have to change snapd itself. This works just fine on macOS & Windows without having to jump through hoops.

[townsend2010]

I'm going to close this since it's solved now.

[jy-metservice]

@wolfenstein87 did you have to make any other changes to get this to work?

I've got a filesystem mounted on /mnt/volumes:

mount |grep volume
/dev/sda1 on /mnt/volumes type ext4 (rw,relatime)

Mount point owned by root:

ls -l /mnt/
total 4
drwxr-xr-x 3 root root 4096 Nov 17 01:28 volumes

I've created the override.conf for multipass:

cat /etc/systemd/system/snap.multipass.multipassd.service.d/override.conf
[Service]
Environment=MULTIPASS_STORAGE=/mnt/volumes/

Have reloaded the systemd ok: systemctl daemon-reload

And attempted to start multipass up again: snap start multipass

But I still get the apparmor errors:

Nov 17 01:30:46 MS008426 kernel: [6745826.183367] audit: type=1400 audit(1668648646.306:7106293): apparmor="DENIED" operation="mkdir" profile="snap.multipass.multipassd" name="/mnt/volumes/cache/" pid=1210539 comm="multipassd" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 17 01:30:46 MS008426 systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 4.
Nov 17 01:30:46 MS008426 systemd[1]: Stopped Service for snap application multipass.multipassd.
Nov 17 01:30:46 MS008426 systemd[1]: Started Service for snap application multipass.multipassd.
Nov 17 01:30:46 MS008426 kernel: [6745826.434647] audit: type=1400 audit(1668648646.554:7106295): apparmor="DENIED" operation="mkdir" profile="snap.multipass.multipassd" name="/mnt/volumes/cache/" pid=1210586 comm="multipassd" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 17 01:30:46 MS008426 multipassd[1210586]: Caught an unhandled exception: unable to create directory '/mnt/volumes/cache'
Nov 17 01:30:46 MS008426 systemd[1]: snap.multipass.multipassd.service: Main process exited, code=exited, status=1/FAILURE
Nov 17 01:30:46 MS008426 systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.

I created the following directories and touched the dnsmasq.leases file

ls -lR |grep -v lost
.:
total 24
drwxr-xr-x 2 root root  4096 Nov 17 01:38 cache
drwxr-xr-x 3 root root  4096 Nov 17 01:38 data

./cache:
total 0

./data:
total 4
drwxr-xr-x 2 root root 4096 Nov 17 01:39 network

./data/network:
total 0
-rw-r--r-- 1 775 root 0 Nov 17 01:39 dnsmasq.leases

But it still finally complains about it:

Nov 17 01:40:50 MS008426 dnsmasq[1212094]: cannot open or create lease file /mnt/volumes/data/network/dnsmasq.leases: Permission denied
Nov 17 01:40:50 MS008426 multipassd[1212043]: #012dnsmasq: cannot open or create lease file /mnt/volumes/data/network/dnsmasq.leases: Permission denied

My / only has 120GB and /dev/sda1 has >1TB available. I want multipass to have access to all 1TB for the VMs etc
Any further pointers appreciated