Use QEMU and "arch-test" to avoid bad binaries in the future#121
Merged
tianon merged 2 commits intotianon:masterfrom Dec 20, 2022
Merged
Use QEMU and "arch-test" to avoid bad binaries in the future#121tianon merged 2 commits intotianon:masterfrom
tianon merged 2 commits intotianon:masterfrom
Conversation
tianon
force-pushed
the
qemu
branch
2 times, most recently
from
3cc30fd to
1960dca
Compare
Owner
Author
|
There's the QEMU bug 😭 (it hangs indefinitely in whatever this failure mode is, hence all the |
Owner
Author
|
There we go, there's our MIPS failure! https://github.com/tianon/gosu/actions/runs/3736241126/jobs/6340334082#step:4:10830 Long Silly Backtrace:+ timeout 1s /go/bin/gosu-mips64el nobody id
trap:5114, a123456=[0,0,0,0,0,0]
trap:5114, a123456=[0,0,0,0,0,0]
results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0
fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted
goroutine 0 [idle]:
results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0
fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc00003fc90, fp:0xc00003fcd8} stack=[0xc000038000,0xc000040000)
0x000000c00003fb90: 0x0000000000000000 0x0000000000000000
0x000000c00003fba0: 0x0000000000000000 0x0000000000000000
0x000000c00003fbb0: 0x0000000000000000 0x0000000000000000
0x000000c00003fbc0: 0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac> 0x0000000000000000
0x000000c00003fbd0: 0x0000000000000000 0x000000c00003fc28
0x000000c00003fbe0: 0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028> 0x0000000000000021
0x000000c00003fbf0: 0x000000c000002340 0x0000000000000000
0x000000c00003fc00: 0x0000000000000000 0x0000000000000000
0x000000c00003fc10: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fc20: 0x000000000008cf74 <runtime.sigtramp+0x0000000000000054> 0x000000c000000021
0x000000c00003fc30: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fc40: 0x000000c0000024e0 0x0000000000000000
0x000000c00003fc50: 0x0000000000000000 0x0000000000000000
0x000000c00003fc60: 0x0000000000000000 0x0000000000000000
0x000000c00003fc70: 0x0000000000000000 0x000000c0000024e0
0x000000c00003fc80: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fc90: <0x0000004001001008 0x0000000000000021
0x000000c00003fca0: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fcb0: 0x0000000000000000 0x0000000000000000
0x000000c00003fcc0: 0x0000000000000000 0x0000000000000000
0x000000c00003fcd0: 0x0000000000000000 >0x0000000000000000
0x000000c00003fce0: 0x0000000000000000 0x0000000000000000
0x000000c00003fcf0: 0xfffffffa00000021 0x0000000000000000
0x000000c00003fd00: 0x00000000000002a0 0x0000000000000000
0x000000c00003fd10: 0x0000000000000000 0x0000000000000000
0x000000c00003fd20: 0x0000000000000000 0x0000000000000000
0x000000c00003fd30: 0x0000000000000000 0x0000000000000000
0x000000c00003fd40: 0x0000000000000000 0x0000000000000000
0x000000c00003fd50: 0x0000000000000000 0x0000000000000000
0x000000c00003fd60: 0x0000000000000000 0x0000000000000000
0x000000c00003fd70: 0x0000000000000000 0x0000000000000000
0x000000c00003fd80: 0x000000c000038000 0x0000000000008000
0x000000c00003fd90: 0x0000000000000000 0x0000000000000000
0x000000c00003fda0: 0x000000c000045ee8 0x0000000000000004
0x000000c00003fdb0: 0x000000003b9aca00 0x000000000026ec38
0x000000c00003fdc0: 0x0000000000000080 0x0000000000000000
0x000000c00003fdd0: 0x0000000000000001
runtime.throw({0x183f3e, 0x44})
/usr/local/go/src/runtime/panic.go:992 +0x6c
runtime.runPerThreadSyscall()
/usr/local/go/src/runtime/os_linux.go:874 +0x410
runtime.sighandler(0x21, 0xc00003fcf0, 0xc00003fd70, 0xc0000024e0)
/usr/local/go/src/runtime/signal_unix.go:631 +0x770
runtime.sigtrampgo(0x21, 0xc00003fcf0, 0xc00003fd70)
/usr/local/go/src/runtime/signal_unix.go:477 +0x228
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc00003fc90, fp:0xc00003fcd8} stack=[0xc000038000,0xc000040000)
0x000000c00003fb90: 0x0000000000000000 0x0000000000000000
0x000000c00003fba0: 0x0000000000000000 0x0000000000000000
0x000000c00003fbb0: 0x0000000000000000 0x0000000000000000
0x000000c00003fbc0: 0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac> 0x0000000000000000
0x000000c00003fbd0: 0x0000000000000000 0x000000c00003fc28
0x000000c00003fbe0: 0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028> 0x0000000000000021
0x000000c00003fbf0: 0x000000c000002340 0x0000000000000000
0x000000c00003fc00: 0x0000000000000000 0x0000000000000000
0x000000c00003fc10: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fc20: 0x000000000008cf74 <runtime.sigtramp+0x0000000000000054> 0x000000c000000021
0x000000c00003fc30: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fc40: 0x000000c0000024e0 0x0000000000000000
0x000000c00003fc50: 0x0000000000000000 0x0000000000000000
0x000000c00003fc60: 0x0000000000000000 0x0000000000000000
0x000000c00003fc70: 0x0000000000000000 0x000000c0000024e0
0x000000c00003fc80: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fc90: <0x0000004001001008 0x0000000000000021
0x000000c00003fca0: 0x000000c00003fcf0 0x000000c00003fd70
0x000000c00003fcb0: 0x0000000000000000 0x0000000000000000
0x000000c00003fcc0: 0x0000000000000000 0x0000000000000000
0x000000c00003fcd0: 0x0000000000000000 >0x0000000000000000
0x000000c00003fce0: 0x0000000000000000 0x0000000000000000
0x000000c00003fcf0: 0xfffffffa00000021 0x0000000000000000
0x000000c00003fd00: 0x00000000000002a0 0x0000000000000000
0x000000c00003fd10: 0x0000000000000000 0x0000000000000000
0x000000c00003fd20: 0x0000000000000000 0x0000000000000000
0x000000c00003fd30: 0x0000000000000000 0x0000000000000000
0x000000c00003fd40: 0x0000000000000000 0x0000000000000000
0x000000c00003fd50: 0x0000000000000000 0x0000000000000000
0x000000c00003fd60: 0x0000000000000000 0x0000000000000000
0x000000c00003fd70: 0x0000000000000000 0x0000000000000000
0x000000c00003fd80: 0x000000c000038000 0x0000000000008000
0x000000c00003fd90: 0x0000000000000000 0x0000000000000000
0x000000c00003fda0: 0x000000c000045ee8 0x0000000000000004
0x000000c00003fdb0: 0x000000003b9aca00 0x000000000026ec38
0x000000c00003fdc0: 0x0000000000000080 0x0000000000000000
0x000000c00003fdd0: 0x0000000000000001
runtime.sigtramp()
/usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54
trap:5114, a123456=[0,0,0,0,0,0]
results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0
fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted
goroutine 1 [running, locked to thread]:
goroutine running on other thread; stack unavailable
goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000057c90, fp:0xc000057cd8} stack=[0xc000050000,0xc000058000)
0x000000c000057b90: 0x0000000000000000 0x0000000000000000
0x000000c000057ba0: 0x0000000000000000 0x0000000000000000
0x000000c000057bb0: 0x0000000000000000 0x0000000000000000
0x000000c000057bc0: 0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac> 0x0000000000000000
0x000000c000057bd0: 0x0000000000000000 0x000000c000057c28
0x000000c000057be0: 0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028> 0x0000000000000021
0x000000c000057bf0: 0x000000c000002b60 0x0000000000000000
0x000000c000057c00: 0x0000000000000000 0x0000000000000000
0x000000c000057c10: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057c20: 0x000000000008cf74 <runtime.sigtramp+0x0000000000000054> 0x000000c000000021
0x000000c000057c30: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057c40: 0x000000c000002d00 0x0000000000000000
0x000000c000057c50: 0x0000000000000000 0x0000000000000000
0x000000c000057c60: 0x0000000000000000 0x0000000000000000
0x000000c000057c70: 0x0000000000000000 0x000000c000002d00
0x000000c000057c80: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057c90: <0x0000004001001008 0x0000000000000021
0x000000c000057ca0: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057cb0: 0x0000000000000000 0x0000000000000000
0x000000c000057cc0: 0x0000000000000000 0x0000000000000000
0x000000c000057cd0: 0x0000000000000000 >0x0000000000000000
0x000000c000057ce0: 0x0000000000000000 0x0000000000000000
0x000000c000057cf0: 0xfffffffa00000021 0x0000000000000000
0x000000c000057d00: 0x00000000000002a0 0x0000000000000000
0x000000c000057d10: 0x0000000000000000 0x0000000000000000
0x000000c000057d20: 0x0000000000000000 0x0000000000000000
0x000000c000057d30: 0x0000000000000000 0x0000000000000000
0x000000c000057d40: 0x0000000000000000 0x0000000000000000
0x000000c000057d50: 0x0000000000000000 0x0000000000000000
0x000000c000057d60: 0x0000000000000000 0x0000000000000000
0x000000c000057d70: 0x0000000000000000 0x0000000000000000
0x000000c000057d80: 0x000000c000050000 0x0000000000008000
0x000000c000057d90: 0x0000000000000000 0x0000000000000000
0x000000c000057da0: 0x0000000000000000 0x000000000000144a
0x000000c000057db0: 0x000000c000002d00 0x000000c000036950
0x000000c000057dc0: 0x0000000000000080 0x0000000000000000
0x000000c000057dd0: 0x0000000000000000
runtime.throw({0x183f3e, 0x44})
/usr/local/go/src/runtime/panic.go:992 +0x6c
runtime.runPerThreadSyscall()
/usr/local/go/src/runtime/os_linux.go:874 +0x410
runtime.sighandler(0x21, 0xc000057cf0, 0xc000057d70, 0xc000002d00)
/usr/local/go/src/runtime/signal_unix.go:631 +0x770
runtime.sigtrampgo(0x21, 0xc000057cf0, 0xc000057d70)
/usr/local/go/src/runtime/signal_unix.go:477 +0x228
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000057c90, fp:0xc000057cd8} stack=[0xc000050000,0xc000058000)
0x000000c000057b90: 0x0000000000000000 0x0000000000000000
0x000000c000057ba0: 0x0000000000000000 0x0000000000000000
0x000000c000057bb0: 0x0000000000000000 0x0000000000000000
0x000000c000057bc0: 0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac> 0x0000000000000000
0x000000c000057bd0: 0x0000000000000000 0x000000c000057c28
0x000000c000057be0: 0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028> 0x0000000000000021
0x000000c000057bf0: 0x000000c000002b60 0x0000000000000000
0x000000c000057c00: 0x0000000000000000 0x0000000000000000
0x000000c000057c10: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057c20: 0x000000000008cf74 <runtime.sigtramp+0x0000000000000054> 0x000000c000000021
0x000000c000057c30: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057c40: 0x000000c000002d00 0x0000000000000000
0x000000c000057c50: 0x0000000000000000 0x0000000000000000
0x000000c000057c60: 0x0000000000000000 0x0000000000000000
0x000000c000057c70: 0x0000000000000000 0x000000c000002d00
0x000000c000057c80: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057c90: <0x0000004001001008 0x0000000000000021
0x000000c000057ca0: 0x000000c000057cf0 0x000000c000057d70
0x000000c000057cb0: 0x0000000000000000 0x0000000000000000
0x000000c000057cc0: 0x0000000000000000 0x0000000000000000
0x000000c000057cd0: 0x0000000000000000 >0x0000000000000000
0x000000c000057ce0: 0x0000000000000000 0x0000000000000000
0x000000c000057cf0: 0xfffffffa00000021 0x0000000000000000
0x000000c000057d00: 0x00000000000002a0 0x0000000000000000
0x000000c000057d10: 0x0000000000000000 0x0000000000000000
0x000000c000057d20: 0x0000000000000000 0x0000000000000000
0x000000c000057d30: 0x0000000000000000 0x0000000000000000
0x000000c000057d40: 0x0000000000000000 0x0000000000000000
0x000000c000057d50: 0x0000000000000000 0x0000000000000000
0x000000c000057d60: 0x0000000000000000 0x0000000000000000
0x000000c000057d70: 0x0000000000000000 0x0000000000000000
0x000000c000057d80: 0x000000c000050000 0x0000000000008000
0x000000c000057d90: 0x0000000000000000 0x0000000000000000
0x000000c000057da0: 0x0000000000000000 0x000000000000144a
0x000000c000057db0: 0x000000c000002d00 0x000000c000036950
0x000000c000057dc0: 0x0000000000000080 0x0000000000000000
0x000000c000057dd0: 0x0000000000000000
runtime.sigtramp()
/usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54
goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000061c90, fp:0xc000061cd8} stack=[0xc00005a000,0xc000062000)
0x000000c000061b90: 0x0000000000000000 0x0000000000000000
0x000000c000061ba0: 0x0000000000000000 0x0000000000000000
0x000000c000061bb0: 0x0000000000000000 0x0000000000000000
0x000000c000061bc0: 0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac> 0x0000000000000000
0x000000c000061bd0: 0x0000000000000000 0x000000c000061c28
0x000000c000061be0: 0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028> 0x0000000000000021
0x000000c000061bf0: 0x000000c000002ea0 0x0000000000000000
0x000000c000061c00: 0x0000000000000000 0x0000000000000000
0x000000c000061c10: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061c20: 0x000000000008cf74 <runtime.sigtramp+0x0000000000000054> 0x000000c000000021
0x000000c000061c30: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061c40: 0x000000c000003040 0x0000000000000000
0x000000c000061c50: 0x0000000000000000 0x0000000000000000
0x000000c000061c60: 0x0000000000000000 0x0000000000000000
0x000000c000061c70: 0x0000000000000000 0x000000c000003040
0x000000c000061c80: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061c90: <0x0000004001001008 0x0000000000000021
0x000000c000061ca0: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061cb0: 0x0000000000000000 0x0000000000000000
0x000000c000061cc0: 0x0000000000000000 0x0000000000000000
0x000000c000061cd0: 0x0000000000000000 >0x0000000000000000
0x000000c000061ce0: 0x0000000000000000 0x0000000000000000
0x000000c000061cf0: 0xfffffffa00000021 0x0000000000000000
0x000000c000061d00: 0x00000000000002a0 0x0000000000000000
0x000000c000061d10: 0x0000000000000000 0x0000000000000000
0x000000c000061d20: 0x0000000000000000 0x0000000000000000
0x000000c000061d30: 0x0000000000000000 0x0000000000000000
0x000000c000061d40: 0x0000000000000000 0x0000000000000000
0x000000c000061d50: 0x0000000000000000 0x0000000000000000
0x000000c000061d60: 0x0000000000000000 0x0000000000000000
0x000000c000061d70: 0x0000000000000000 0x0000000000000000
0x000000c000061d80: 0x000000c00005a000 0x0000000000008000
0x000000c000061d90: 0x0000000000000000 0x0000000000000000
0x000000c000061da0: 0x0000000000000000 0x000000000000144a
0x000000c000061db0: 0x000000c000003040 0x000000000029b098
0x000000c000061dc0: 0x0000000000000080 0x0000000000000000
0x000000c000061dd0: 0x0000000000000000
runtime.throw({0x183f3e, 0x44})
/usr/local/go/src/runtime/panic.go:992 +0x6c
runtime.runPerThreadSyscall()
/usr/local/go/src/runtime/os_linux.go:874 +0x410
runtime.sighandler(0x21, 0xc000061cf0, 0xc000061d70, 0xc000003040)
/usr/local/go/src/runtime/signal_unix.go:631 +0x770
runtime.sigtrampgo(0x21, 0xc000061cf0, 0xc000061d70)
/usr/local/go/src/runtime/signal_unix.go:477 +0x228
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000061c90, fp:0xc000061cd8} stack=[0xc00005a000,0xc000062000)
0x000000c000061b90: 0x0000000000000000 0x0000000000000000
0x000000c000061ba0: 0x0000000000000000 0x0000000000000000
0x000000c000061bb0: 0x0000000000000000 0x0000000000000000
0x000000c000061bc0: 0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac> 0x0000000000000000
0x000000c000061bd0: 0x0000000000000000 0x000000c000061c28
0x000000c000061be0: 0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028> 0x0000000000000021
0x000000c000061bf0: 0x000000c000002ea0 0x0000000000000000
0x000000c000061c00: 0x0000000000000000 0x0000000000000000
0x000000c000061c10: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061c20: 0x000000000008cf74 <runtime.sigtramp+0x0000000000000054> 0x000000c000000021
0x000000c000061c30: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061c40: 0x000000c000003040 0x0000000000000000
0x000000c000061c50: 0x0000000000000000 0x0000000000000000
0x000000c000061c60: 0x0000000000000000 0x0000000000000000
0x000000c000061c70: 0x0000000000000000 0x000000c000003040
0x000000c000061c80: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061c90: <0x0000004001001008 0x0000000000000021
0x000000c000061ca0: 0x000000c000061cf0 0x000000c000061d70
0x000000c000061cb0: 0x0000000000000000 0x0000000000000000
0x000000c000061cc0: 0x0000000000000000 0x0000000000000000
0x000000c000061cd0: 0x0000000000000000 >0x0000000000000000
0x000000c000061ce0: 0x0000000000000000 0x0000000000000000
0x000000c000061cf0: 0xfffffffa00000021 0x0000000000000000
0x000000c000061d00: 0x00000000000002a0 0x0000000000000000
0x000000c000061d10: 0x0000000000000000 0x0000000000000000
0x000000c000061d20: 0x0000000000000000 0x0000000000000000
0x000000c000061d30: 0x0000000000000000 0x0000000000000000
0x000000c000061d40: 0x0000000000000000 0x0000000000000000
0x000000c000061d50: 0x0000000000000000 0x0000000000000000
0x000000c000061d60: 0x0000000000000000 0x0000000000000000
0x000000c000061d70: 0x0000000000000000 0x0000000000000000
0x000000c000061d80: 0x000000c00005a000 0x0000000000008000
0x000000c000061d90: 0x0000000000000000 0x0000000000000000
0x000000c000061da0: 0x0000000000000000 0x000000000000144a
0x000000c000061db0: 0x000000c000003040 0x000000000029b098
0x000000c000061dc0: 0x0000000000000080 0x0000000000000000
0x000000c000061dd0: 0x0000000000000000
runtime.sigtramp()
/usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54 |
This fixes our `mips64le` builds.
Owner
Author
|
A nice benefit of container-based builds: we can just apply that upstream patch to our Go sources during build and the bug is now fixed. 😂 |
Collaborator
|
The (Just to post this somewhere since fixing it in Debian Bookworm would be unlikely since it would need a patch applied to the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As now noted on https://github.com/tianon/gosu/releases/tag/1.15, our new releases are affected by golang/go#56426, which is broken in Go 1.18+ and not fixed until Go 1.20. 😞
This is an attempt to pre-test for that via QEMU user-mode emulation and actually testing our end binaries (because this reproduces the failure).
Another failure that I think this is also going to run into is that from Go 1.18+ it seems our binaries now often hang in QEMU user-mode emulation (most reliably on ARM), so that's fun.