Test Case Config does not behave as I expected

[Jager567]

The documentation states that I can add a timeout to a test. It states that it is "Useful for code that is non-deterministic and might not finish". This leads me to believe that this setting will stop the test execution after the given duration has passed. However, when I accidentally executed a test that called a method with an infinite loop, it didn't stop the execution after the timeout had passed. Is this a documentation issue, a fault on my side or a bug?

After some testing, I found out that this fails, but only after the full test has finished:

class FiniteLoopString: StringSpec() {
    "has a long finite loop test".config(timeout = 100.milliseconds) {
        val startTime = currentTimeMillis()
        while (currentTimeMillis() < startTime + 1000) {
            "this" shouldNotBe "that"
        }
    }
}

while the following test just keeps running forever:

class InfiniteLoopString: StringSpec() {
    "has an infinite loop test".config(timeout = 100.milliseconds) {
        while (true) {
            "this" shouldNotBe "that"
        }
    }
}

During my testing, I also found that this test succeeds, even though it takes longer than the timeout:

class SuccessfulTest: WordSpec() {
    init {
        "String.length" should {
            "return the length of the string".config(timeout = 2.seconds) {
                val startTime = currentTimeMillis()
                while (currentTimeMillis() < startTime + 10000) {
                    "this" shouldNotBe "that"
                }
            }
        }
    }
}

[sksamuel]

Definitely a bug. The timeout should kick in for any test that takes longer than the timeout.

[sksamuel]

Which version of kotlintest ?

[Jager567]

I tested this with version 3.1.10.

[LeoColman]

I'll grab this if you're not working on it, @sksamuel .

[sksamuel]

I'll fix this as part of #518 as it will need to work differently.

[sksamuel]

Added. Review as part of #518

[sksamuel]

Please try 3.2.0-RC1 as this should be fixed now. @Jager567

[Jager567]

It seems that test cases now fail correctly, but they still aren't aborted when they haven't finished executing within the timeout period. This means that

class SuccessfulTest: WordSpec() {
	init {
		"String.length" should {
			"return the length of the string".config(timeout = 2.seconds) {
				val startTime = currentTimeMillis()
				while (currentTimeMillis() < startTime + 10000) {
					"this" shouldNotBe "that"
				}
			}
		}
	}
}

now correctly fails, but only after having executed for 10 seconds, while a test with an infinite loop will never finish.