Build on Github Action with macos-latest missing dependencies

[zonyitoo]

Problem

https://github.com/shadowsocks/shadowsocks-rust/runs/962064929?check_suite_focus=true

Build running on Github Action with macos-latest always reports missing serde_derive.

• serde_derive should already be included by serde's feature derive: https://github.com/serde-rs/serde/blob/53b9871b172c1c29a16e66138f108bf4155444fe/serde/Cargo.toml#L35
• shadowsocks-rust's Cargo.toml have specified the derive feature: https://github.com/shadowsocks/shadowsocks-rust/blob/6451f33b2e9471b44feb315778d6f60d81634a86/Cargo.toml#L116
• It only happens in Github Action's macos-latest runner. I can't reproduce it locally.

[zonyitoo]

It works if cache is disabled.

[ehuss]

Without the cache, this will be very difficult to diagnose. The most likely cause that I can think of is that the libserde_derive.dylib file was truncated or otherwise corrupted in the cache. In that circumstance, cargo doesn't know it needs to be rebuilt, and rustc will silently ignore the dylib. Unfortunately I can't think of much that can be done in that situation, other than clearing the cache.

If you're able to reproduce on CI, please let me know!

[zonyitoo]

It is reproduciable on CI nearly 100% just by enabling cache.

[ehuss]

So I've been digging into this a little bit, and I've been able to reproduce on GHA. Unfortunately I cannot reproduce it locally. As far as I see, it looks to be an issue with the GitHub cache action. After clearing the cache, and running cargo build in your project, I added some debugging lines and discovered that the cache is overwriting 0's to the first 8MB (0x800000 bytes) of the target/debug/deps/libserde_derive-797b01cb80d42716.dylib file.

I tried reproducing locally using the tar commands displayed in the build logs to reproduce their caching behavior, but I could not reproduce the problem.

The problem also seems to be very sensitive to the nature of your project. Removing the RUSTFLAGS: "-Ctarget-feature=+aes,+ssse3" makes the problem go away, as well as removing --features "aes-pmac-siv". I have no idea why this would affect things.

I would recommend you follow up with GitHub on their issue tracker (https://github.com/actions/cache) to figure out why the cache is corrupting that file.

Here's a basic example from one of my tests:

1. From a cleared cache: https://github.com/ehuss/shadowsocks-rust/runs/1004677582?check_suite_focus=true
   The last step shows the shasum of the file: 0aeb971a8830e77ee92843e42e424887a448862e target/debug/deps/libserde_derive-797b01cb80d42716.dylib
2. The next run, which makes a small modification to src/lib.rs: https://github.com/ehuss/shadowsocks-rust/runs/1004692842?check_suite_focus=true
   Immediately after the cache has been extracted, the shasum of the file is: 4e2ef1b28a57bd1c72f5df110b800dcbf8d472d1 target/debug/deps/libserde_derive-797b01cb80d42716.dylib

I think this pretty clearly shows the cache is modifying it somehow. I did some other tests where I downloaded the file, and confirmed that the first 8MB of the file has been zeroed out.

[dhadka]

@ehuss This is a really weird one. I put some more details over in actions/cache#403, but the summary is that:

1. I can reproduce the issue by just running tar -cf ... followed by tar -xf ..., so it seems unrelated to the cache and instead happens during tar
2. Depending on what other commands I'm running before creating the tar, the issue magically disappears. For example, if I add sleep 10 or just add verbose logging to tar (tar -cvf ...) I can't repro.

[bk2204]

If this is Catalina, I think this may be the same bug as Homebrew/brew#6539. (I'm familiar with this because I maintain Git LFS and it broke that Homebrew package.) That bug mentions a workaround, so it may be worth trying to figure out (or ask) what the Homebrew folks did and see if that might work for you.

[ehuss]

I think the workaround they are referencing is calling /usr/sbin/purge (as root), which flushes the disk cache.

Another workaround is to use GNU tar as described here: actions/cache#403 (comment)

I wonder if there is some strange interaction with APFS sparse files.