Skip to content

Commit 76a2074

Browse files
qinqononsi
authored andcommitted
feat: add --sleep-on-failure to pause a failed spec before teardown
When a spec fails against a live environment, its teardown (AfterEach/JustAfterEach/DeferCleanup) tears down the very state needed to debug the failure. --sleep-on-failure=<duration> pauses the suite the moment a failure is identified - before any teardown runs - so the live system can be inspected. The pause hooks directly into the suite's existing failure-handling path (runNode): when a node finishes with a failure, Ginkgo emits the failure and then, if the flag is set, pauses right there. Only failures in setup and subject nodes (It, Before*, BeforeSuite...) pause; failures in teardown/cleanup or reporting nodes do not, since the system is already being torn down at that point. The pause is interruptible: pressing ^C (or any interrupt) ends the pause early and the suite proceeds to run cleanup as usual rather than skipping it. This is a debugging aid for interactive, serial runs. Because Ginkgo's parallelism is multi-process, a single failing spec cannot meaningfully freeze the whole system, so combining --sleep-on-failure with -p/--procs is rejected with a configuration error. Adds unit (internal_integration) and CLI (integration) tests and docs. Assisted-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Enrique Llorente <ellorent@redhat.com>
1 parent 3c7bde4 commit 76a2074

8 files changed

Lines changed: 340 additions & 1 deletion

File tree

docs/index.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3334,6 +3334,24 @@ One final, somewhat complex, note on timeouts and the Grace Period. As mentione
33343334
- If the remaining node is interruptible and **does not** have a `NodeTimeout`, Ginkgo uses the Grace Period to set a deadline for the node. If the deadline expires, then a second Grace Period applies before Ginkgo leaks the node and moves on.
33353335
- If the remaining node is **not** interruptible, Ginkgo will give the node a single Grace Period to complete and exit. In this case, since it cannot be interrupted, Ginkgo will simply leak the node after one Grace Period.
33363336

3337+
#### Pausing a Failed Spec for Debugging: The --sleep-on-failure flag
3338+
3339+
When a spec fails against a real environment (a cluster, database, or service) the spec's teardown - its `AfterEach`, `JustAfterEach`, and `DeferCleanup` nodes - typically tears down the very state you need to inspect to understand the failure. By the time you can attach a debugger or run `kubectl`/`psql`, the evidence is gone.
3340+
3341+
The `--sleep-on-failure=<DURATION>` flag pauses a spec **after it fails but before its teardown runs**, leaving the system live so you can perform forensics:
3342+
3343+
```bash
3344+
ginkgo --sleep-on-failure=30m ./...
3345+
```
3346+
3347+
With this flag set, the instant a spec fails Ginkgo emits the failure and then pauses for the configured duration before continuing. The pause happens at the point of failure - before any teardown (`AfterEach`/`JustAfterEach`/`DeferCleanup`) runs - so the failing spec's resources remain in place for you to inspect. Ginkgo emits a progress report announcing the pause so you know the suite is waiting and what to do next. Failures that occur in setup and subject nodes (`It`, `BeforeEach`, `BeforeAll`, ...) trigger the pause; failures inside teardown nodes do not, since the system is already being torn down at that point.
3348+
3349+
The pause is interruptible: pressing `^C` (or otherwise interrupting the suite) ends the pause early and proceeds to run the spec's cleanup nodes - it does not skip teardown. This means you can pause indefinitely (e.g. `--sleep-on-failure=24h`), inspect the system at your leisure, and then press `^C` once to resume and let Ginkgo clean up.
3350+
3351+
Specs that pass are never paused, and the flag has no effect when set to `0` (the default).
3352+
3353+
`--sleep-on-failure` is a debugging aid intended for interactive, serial runs and is **only supported in serial mode**. Because Ginkgo's parallelism is multi-process, a single failing spec cannot meaningfully freeze the whole system, so combining `--sleep-on-failure` with `-p`/`--procs` is rejected with a configuration error. Run the specific failing spec serially (for example with `--focus` and without `-p`) when you want to use it.
3354+
33373355
#### Using SpecContext with Gomega's Eventually
33383356

33393357
Gomega provides `Eventually` to allow you to poll an object or function repeatedly until a Gomega matcher is satisfied. `Eventually` integrates cleanly with interruptible nodes by accepting a `SpecContext`/`context.Context` parameter. This allows you, for example, to enforce a single timeout across a set of polling assertions:
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
package sleep_on_failure_fixture_test
2+
3+
import (
4+
"testing"
5+
6+
. "github.com/onsi/ginkgo/v2"
7+
. "github.com/onsi/gomega"
8+
)
9+
10+
func TestSleepOnFailureFixture(t *testing.T) {
11+
RegisterFailHandler(Fail)
12+
RunSpecs(t, "SleepOnFailure Fixture Suite")
13+
}
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
package sleep_on_failure_fixture_test
2+
3+
import (
4+
"fmt"
5+
"os"
6+
7+
. "github.com/onsi/ginkgo/v2"
8+
)
9+
10+
var _ = Describe("sleep on failure", func() {
11+
It("passes and is never paused", func() {
12+
fmt.Fprintln(os.Stdout, "PASSING-SPEC-RAN")
13+
})
14+
15+
It("fails and should be paused before teardown", func() {
16+
fmt.Fprintln(os.Stdout, "FAILING-SPEC-BODY-RAN")
17+
Fail("intentional failure")
18+
})
19+
20+
AfterEach(func() {
21+
fmt.Fprintln(os.Stdout, "TEARDOWN-RAN")
22+
})
23+
})
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
package integration_test
2+
3+
import (
4+
"time"
5+
6+
. "github.com/onsi/ginkgo/v2"
7+
. "github.com/onsi/gomega"
8+
"github.com/onsi/gomega/gexec"
9+
)
10+
11+
var _ = Describe("--sleep-on-failure", func() {
12+
BeforeEach(func() {
13+
fm.MountFixture("sleep_on_failure")
14+
})
15+
16+
Context("when running serially with --sleep-on-failure set", func() {
17+
It("pauses a failed spec before teardown, then proceeds to teardown", func() {
18+
start := time.Now()
19+
session := startGinkgo(fm.PathTo("sleep_on_failure"), "--no-color", "--sleep-on-failure=2s")
20+
Eventually(session).Should(gexec.Exit(1))
21+
elapsed := time.Since(start)
22+
output := string(session.Out.Contents())
23+
24+
// it actually waited for (about) the configured duration
25+
Ω(elapsed).Should(BeNumerically(">=", 1500*time.Millisecond), "should have paused for ~2s")
26+
27+
// it announced the pause and what to do
28+
Ω(output).Should(ContainSubstring("Paused on failure"))
29+
30+
// it paused before teardown: the failing body runs, then the pause, then teardown
31+
Ω(output).Should(MatchRegexp(`(?s)FAILING-SPEC-BODY-RAN.*Paused on failure.*TEARDOWN-RAN`),
32+
"the pause must occur after the failing spec body and before its teardown")
33+
34+
// teardown still ran after the pause
35+
Ω(output).Should(ContainSubstring("TEARDOWN-RAN"))
36+
})
37+
38+
It("does not pause specs that pass", func() {
39+
session := startGinkgo(fm.PathTo("sleep_on_failure"), "--no-color", "--sleep-on-failure=1h", "--focus=passes and is never paused")
40+
// if the passing spec were paused, this would hang for an hour; the suite timeout/Eventually guards us
41+
Eventually(session, 30*time.Second).Should(gexec.Exit(0))
42+
output := string(session.Out.Contents())
43+
44+
Ω(output).Should(ContainSubstring("PASSING-SPEC-RAN"))
45+
Ω(output).ShouldNot(ContainSubstring("Paused on failure"))
46+
})
47+
})
48+
49+
Context("when running in parallel with --sleep-on-failure set", func() {
50+
It("exits with a helpful error instead of pausing", func() {
51+
start := time.Now()
52+
session := startGinkgo(fm.PathTo("sleep_on_failure"), "--no-color", "--procs=2", "--sleep-on-failure=1h")
53+
Eventually(session).Should(gexec.Exit(1))
54+
elapsed := time.Since(start)
55+
output := string(session.Out.Contents()) + string(session.Err.Contents())
56+
57+
// it must not have paused (would have hung for an hour)
58+
Ω(elapsed).Should(BeNumerically("<", time.Minute))
59+
60+
// it explains the serial-only restriction
61+
Ω(output).Should(ContainSubstring("Ginkgo only supports --sleep-on-failure in serial mode"))
62+
Ω(output).ShouldNot(ContainSubstring("Paused on failure"))
63+
})
64+
})
65+
})
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
package internal_integration_test
2+
3+
import (
4+
"time"
5+
6+
. "github.com/onsi/ginkgo/v2"
7+
. "github.com/onsi/gomega"
8+
9+
"github.com/onsi/ginkgo/v2/internal/interrupt_handler"
10+
. "github.com/onsi/ginkgo/v2/internal/test_helpers"
11+
)
12+
13+
var _ = Describe("--sleep-on-failure", func() {
14+
Describe("when a spec fails", func() {
15+
BeforeEach(func() {
16+
conf.SleepOnFailure = 50 * time.Millisecond
17+
success, _ := RunFixture("sleep on failure - failing spec", func() {
18+
Context("container", func() {
19+
BeforeEach(rt.T("bef"))
20+
It("A", rt.T("A", func() {
21+
DeferCleanup(rt.T("cleanup"))
22+
F("boom", cl)
23+
}))
24+
JustAfterEach(rt.T("just-after"))
25+
AfterEach(rt.T("aft"))
26+
})
27+
})
28+
Ω(success).Should(BeFalse())
29+
})
30+
31+
It("pauses at the moment of failure, before any teardown runs", func() {
32+
// The pause is hooked into runNode's failure path, so it happens the instant
33+
// the It fails - before any JustAfterEach/AfterEach/DeferCleanup nodes run.
34+
// Teardown then proceeds in the normal order once the pause elapses.
35+
Ω(rt).Should(HaveTracked("bef", "A", "just-after", "aft", "cleanup"))
36+
})
37+
38+
It("still runs teardown and cleanup after the pause", func() {
39+
Ω(rt).Should(HaveRun("aft"))
40+
Ω(rt).Should(HaveRun("cleanup"))
41+
})
42+
43+
It("emits the failure and then a progress report announcing the pause", func() {
44+
Ω(reporter.Did.Find("A")).Should(HaveFailed("boom", cl))
45+
Ω(reporter.ProgressReports).ShouldNot(BeEmpty())
46+
Ω(reporter.ProgressReports[0].Message).Should(ContainSubstring("Paused on failure"))
47+
})
48+
})
49+
50+
Describe("when a failure occurs in a BeforeEach", func() {
51+
BeforeEach(func() {
52+
conf.SleepOnFailure = 50 * time.Millisecond
53+
success, _ := RunFixture("sleep on failure - failing beforeeach", func() {
54+
Context("container", func() {
55+
BeforeEach(rt.T("bef", func() {
56+
F("boom", cl)
57+
}))
58+
It("A", rt.T("A"))
59+
AfterEach(rt.T("aft"))
60+
})
61+
})
62+
Ω(success).Should(BeFalse())
63+
})
64+
65+
It("pauses on the setup failure before teardown, and skips the It", func() {
66+
Ω(rt).Should(HaveTracked("bef", "aft"))
67+
Ω(reporter.ProgressReports).ShouldNot(BeEmpty())
68+
Ω(reporter.ProgressReports[0].Message).Should(ContainSubstring("Paused on failure"))
69+
})
70+
})
71+
72+
Describe("when a spec passes", func() {
73+
var start time.Time
74+
BeforeEach(func() {
75+
conf.SleepOnFailure = time.Hour // would hang if it ever fired on success
76+
start = time.Now()
77+
success, _ := RunFixture("sleep on failure - passing spec", func() {
78+
It("A", rt.T("A"))
79+
AfterEach(rt.T("aft"))
80+
})
81+
Ω(success).Should(BeTrue())
82+
})
83+
84+
It("does not pause and emits no pause progress report", func() {
85+
Ω(time.Since(start)).Should(BeNumerically("<", time.Second))
86+
Ω(rt).Should(HaveTracked("A", "aft"))
87+
Ω(reporter.ProgressReports).Should(BeEmpty())
88+
})
89+
})
90+
91+
Describe("when a failure occurs in teardown", func() {
92+
var start time.Time
93+
BeforeEach(func() {
94+
conf.SleepOnFailure = time.Hour // would hang if the teardown failure paused
95+
start = time.Now()
96+
success, _ := RunFixture("sleep on failure - failing teardown", func() {
97+
It("A", rt.T("A"))
98+
AfterEach(rt.T("aft", func() {
99+
F("boom", cl)
100+
}))
101+
})
102+
Ω(success).Should(BeFalse())
103+
})
104+
105+
It("does not pause (teardown is already running, nothing to inspect)", func() {
106+
Ω(time.Since(start)).Should(BeNumerically("<", time.Second))
107+
Ω(rt).Should(HaveTracked("A", "aft"))
108+
Ω(reporter.ProgressReports).Should(BeEmpty())
109+
})
110+
})
111+
112+
Describe("when the feature is disabled (duration is zero)", func() {
113+
BeforeEach(func() {
114+
conf.SleepOnFailure = 0
115+
success, _ := RunFixture("sleep on failure - disabled", func() {
116+
It("A", rt.T("A", func() {
117+
F("boom", cl)
118+
}))
119+
AfterEach(rt.T("aft"))
120+
})
121+
Ω(success).Should(BeFalse())
122+
})
123+
124+
It("does not pause and emits no pause progress report", func() {
125+
Ω(rt).Should(HaveTracked("A", "aft"))
126+
Ω(reporter.ProgressReports).Should(BeEmpty())
127+
})
128+
})
129+
130+
Describe("when the user interrupts during the pause", func() {
131+
var start time.Time
132+
BeforeEach(func() {
133+
conf.SleepOnFailure = time.Hour // long enough that only the interrupt can end it
134+
start = time.Now()
135+
success, _ := RunFixture("sleep on failure - interrupted", func() {
136+
Context("container", func() {
137+
It("A", rt.T("A", func() {
138+
// fire the interrupt shortly after this node's body returns, so it
139+
// lands while the suite is paused waiting on the failure
140+
go func() {
141+
time.Sleep(100 * time.Millisecond)
142+
interruptHandler.Interrupt(interrupt_handler.InterruptCauseSignal)
143+
}()
144+
F("boom", cl)
145+
}))
146+
AfterEach(rt.T("aft"))
147+
})
148+
})
149+
Ω(success).Should(BeFalse())
150+
})
151+
152+
It("ends the pause early and proceeds to run teardown", func() {
153+
Ω(time.Since(start)).Should(BeNumerically("<", time.Minute))
154+
Ω(rt).Should(HaveTracked("A", "aft"))
155+
})
156+
})
157+
})

internal/suite.go

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -996,6 +996,7 @@ func (suite *Suite) runNode(node Node, specDeadline time.Time, text string) (typ
996996
} else {
997997
failure.Message, failure.Location, failure.ForwardedPanic, failure.TimelineLocation = failureFromRun.Message, failureFromRun.Location, failureFromRun.ForwardedPanic, failureFromRun.TimelineLocation
998998
suite.reporter.EmitFailure(outcomeFromRun, failure)
999+
suite.pauseOnFailureIfRequested(node)
9991000
return outcomeFromRun, failure
10001001
}
10011002
case <-gracePeriodChannel:
@@ -1079,6 +1080,42 @@ func (suite *Suite) runNode(node Node, specDeadline time.Time, text string) (typ
10791080
}
10801081
}
10811082

1083+
// pauseOnFailureIfRequested pauses the suite at the moment a failure is identified,
1084+
// when the user has set --sleep-on-failure. This hooks directly into runNode's failure
1085+
// path so the pause happens immediately at the point of failure - before any teardown
1086+
// or cleanup runs - leaving the system live for inspection.
1087+
//
1088+
// We only pause for failures in setup and subject nodes (It, Before*, BeforeSuite...),
1089+
// i.e. nodes that run before teardown. Pausing on a failure in a teardown/cleanup or
1090+
// reporting node would be pointless (the system is already being torn down) and could
1091+
// interfere with interrupt handling, so those are skipped.
1092+
//
1093+
// The pause is interruptible: pressing ^C (or any interrupt) ends the pause early and
1094+
// the suite proceeds to run cleanup as usual. It is a no-op if the feature is disabled.
1095+
func (suite *Suite) pauseOnFailureIfRequested(node Node) {
1096+
if suite.config.SleepOnFailure <= 0 {
1097+
return
1098+
}
1099+
// only pause before teardown - skip teardown/cleanup/reporting nodes
1100+
if node.NodeType.Is(types.NodeTypesAllowedDuringCleanupInterrupt | types.NodeTypesAllowedDuringReportInterrupt) {
1101+
return
1102+
}
1103+
1104+
duration := suite.config.SleepOnFailure
1105+
report := suite.generateProgressReport(false)
1106+
report.Message = fmt.Sprintf("{{bold}}{{orange}}Paused on failure for up to %s.{{/}}\nThe spec failed and Ginkgo has paused before running any teardown so you can inspect the live system.\nPress {{bold}}^C{{/}} to end the pause and proceed to cleanup.", duration)
1107+
suite.emitProgressReport(report)
1108+
1109+
timer := time.NewTimer(duration)
1110+
defer timer.Stop()
1111+
// wait for the pause to elapse, or for the user to interrupt - in which case we end
1112+
// the pause early and let runNode return so cleanup can proceed
1113+
select {
1114+
case <-timer.C:
1115+
case <-suite.interruptHandler.Status().Channel:
1116+
}
1117+
}
1118+
10821119
// TODO: search for usages and consider if reporter.EmitFailure() is necessary
10831120
func (suite *Suite) failureForLeafNodeWithMessage(node Node, message string) types.Failure {
10841121
return types.Failure{

types/config.go

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ type SuiteConfig struct {
3838
OutputInterceptorMode string
3939
SourceRoots []string
4040
GracePeriod time.Duration
41+
SleepOnFailure time.Duration
4142

4243
ParallelProcess int
4344
ParallelTotal int
@@ -293,6 +294,8 @@ var SuiteConfigFlags = GinkgoFlags{
293294
Usage: "Make up to this many attempts to run each spec. If any of the attempts succeed, the suite will not be failed."},
294295
{KeyPath: "S.FailOnEmpty", Name: "fail-on-empty", SectionKey: "failure",
295296
Usage: "If set, ginkgo will mark the test suite as failed if no specs are run."},
297+
{KeyPath: "S.SleepOnFailure", Name: "sleep-on-failure", SectionKey: "failure", UsageDefaultValue: "0 - disabled",
298+
Usage: "If set, ginkgo will pause for this duration after a spec fails - before its teardown (AfterEach/JustAfterEach/DeferCleanup) runs - so you can inspect the live system. Press ^C to end the pause early and proceed to cleanup. Serial only: cannot be combined with -p/--procs."},
296299

297300
{KeyPath: "S.DryRun", Name: "dry-run", SectionKey: "debug", DeprecatedName: "dryRun", DeprecatedDocLink: "changed-command-line-flags",
298301
Usage: "If set, ginkgo will walk the test hierarchy without actually running anything. Best paired with -v."},
@@ -429,6 +432,14 @@ func VetConfig(flagSet GinkgoFlagSet, suiteConfig SuiteConfig, reporterConfig Re
429432
errors = append(errors, GinkgoErrors.GracePeriodCannotBeZero())
430433
}
431434

435+
if suiteConfig.SleepOnFailure < 0 {
436+
errors = append(errors, GinkgoErrors.InvalidSleepOnFailureConfiguration())
437+
}
438+
439+
if suiteConfig.SleepOnFailure > 0 && suiteConfig.ParallelTotal > 1 {
440+
errors = append(errors, GinkgoErrors.SleepOnFailureInParallelConfiguration())
441+
}
442+
432443
if len(suiteConfig.FocusFiles) > 0 {
433444
_, err := ParseFileFilters(suiteConfig.FocusFiles)
434445
if err != nil {

0 commit comments

Comments
 (0)