runtime: hangs in TestGdbBacktrace on linux

[bcmills]

2020-02-22T04:31:41-059a5ac/linux-mips64le-mengzhuo

goroutine 23401 [syscall, 11 minutes]:
syscall.Syscall6(0x1475, 0x1, 0x3b6f, 0xc000cad968, 0x1000004, 0x0, 0x0, 0x120153878, 0xc000cad960, 0x120080418)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/syscall/asm_linux_mips64x.s:40 +0x10 fp=0xc000cad910 sp=0xc000cad908 pc=0x1200cc678
os.(*Process).blockUntilWaitable(0xc00001ab70, 0x0, 0x1200103ec, 0x0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/os/wait_waitid.go:31 +0x88 fp=0xc000cad9f8 sp=0xc000cad910 pc=0x1200e49e8
os.(*Process).wait(0xc00001ab70, 0x1202d17d0, 0x1202d17d8, 0x1202d17c8)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/os/exec_unix.go:22 +0x4c fp=0xc000cada68 sp=0xc000cad9f8 pc=0x1200df43c
os.(*Process).Wait(...)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc000ad8f20, 0x0, 0x0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/os/exec/exec.go:502 +0x68 fp=0xc000cadad8 sp=0xc000cada68 pc=0x120153e18
os/exec.(*Cmd).Run(0xc000ad8f20, 0xc00009aff0, 0xc000ad8f20)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/os/exec/exec.go:340 +0x74 fp=0xc000cadaf8 sp=0xc000cadad8 pc=0x12015340c
os/exec.(*Cmd).CombinedOutput(0xc000ad8f20, 0x3, 0xc000cade78, 0xf, 0xf, 0xc000ad8f20)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/os/exec/exec.go:562 +0xbc fp=0xc000cadb20 sp=0xc000cadaf8 pc=0x1201541dc
runtime_test.TestGdbBacktrace(0xc00022a5a0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/runtime/runtime-gdb_test.go:388 +0x6c4 fp=0xc000cadf80 sp=0xc000cadb20 pc=0x120202e64
testing.tRunner(0xc00022a5a0, 0x1202d2e00)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/testing/testing.go:992 +0xf8 fp=0xc000cadfc8 sp=0xc000cadf80 pc=0x12010a978
runtime.goexit()
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc000cadfc8 sp=0xc000cadfc8 pc=0x120084354
created by testing.(*T).Run
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/testing/testing.go:1043 +0x378

CC @dr2chase @aclements @mengzhuo

[mengzhuo]

It shows that runtime test failed due to timeout.

FAIL	runtime	720.011s
FAIL

Maybe we should dump ps ef and system load after bots failed?

[gopherbot]

Change https://golang.org/cl/227811 mentions this issue: runtime: only shrink stack while preempting

[bcmills]

This turns out not to be specific to the mips64le builder. See also #39228 (occasional failures instead of hangs).

2021-01-23T19:46:06-9897655/linux-amd64-sid
2020-11-02T03:03:16-0387bed/linux-386-softfloat
2020-06-25T12:02:38-334752d/linux-amd64-staticlockranking

[bcmills]

'(?m)panic: test timed out(?:.|\n)*os/exec\...Cmd.*\n.*\nruntime_test\.TestGdbBacktrace', since 2021-01-23:

2021-07-19T13:27:46-49402be/linux-amd64-staticlockranking
2021-05-17T17:46:08-b9b2bed/linux-amd64-race
2021-04-21T15:57:05-f9e2dbb/linux-amd64-jessie
2021-03-24T14:20:32-747f426/linux-amd64-regabi
2021-03-02T20:28:32-b65091c/linux-amd64-jessie
2021-02-23T15:03:26-a51daac/linux-amd64-race

[bcmills]

greplogs --dashboard -md -l -e '(?m)panic: test timed out(?:.|\n)*os/exec\...Cmd.*\n.*\nruntime_test\.TestGdbBacktrace' --since=2021-07-20

2021-10-13T15:11:16-0454d73/linux-s390x-ibm
2021-10-12T18:52:28-26b6833/linux-386-softfloat
2021-09-27T18:57:20-3d795ea/linux-amd64
2021-08-26T17:19:04-166b691/linux-386-softfloat
2021-08-12T19:06:51-46fd547/linux-amd64-noregabi

[bcmills]

greplogs --dashboard -md -l -e '(?m)panic: test timed out(?:.|\n)*os/exec\...Cmd.*\n.*\nruntime_test\.TestGdbBacktrace' --since=2021-10-15

2021-12-08T04:14:00-a19e72c/linux-amd64-buster
2021-11-16T17:13:33-29ec902/linux-386-softfloat
2021-11-13T00:49:51-530e320/solaris-amd64-oraclerel
2021-10-25T20:08:25-252324e/linux-386-stretch

[bcmills]

Marking as release-blocker for Go 1.18 due to the regularity of failures, and especially the variety of builders on which failures have been observed. The large number of affected platforms suggests either a problem in the test (which we should fix) or a deep-rooted bug in gdb itself (for which we should either add a workaround or remove the infeasible test).

Note that many of the failures are on linux/386 and linux/amd64, which are first-class ports.

(CC @jeremyfaller)