testing: prove is obsolete and archived

[digikar99]

The author has archived Prove and The Common Lisp Wiki mentions it to be obsolete.

Guess we need a newer recipe. But more importantly I think, prove needs to be taken down from the cookbook. Either that, or seeing the number of stars on prove and fiveam, prove needs a new maintainer. Are there any download statistics available?

Edit: Download statistics are available on quicklisp blog for January and February 2018. These suggest prove was fast gaining popularity and fiveam losing. More current statistics would be useful.

[sheepduke]

Rove should be the successor of Prove.

However I have observed a strange behavior (probably a bug) that makes tests fail randomly.

fiveAM seems to be more stable and DWIM.

[vindarel]

+1 we should at least add a warning before someone updates the recipe.

I'd be in favor of Rove. In my (short) experience, FiveAM doesn't play nice with ASDF and the host system, hence a CI system, which I expect Rove to do better. Prove did. (see https://github.com/rpgoldman/fiveam-asdf/)

[sheepduke]

I'd be in favor of Rove.

I experienced some random bugs while using Rove, in the following cases:

1. Using dexador to test HTTP server.
2. Testing functions with handler-case.

Especially the second case, it succeeds at the first time of running (asdf:test-system), but fails later.

I am in vocation and cannot provide exact code example. I will be back next week.

So I consider it as less stable compared with FiveAM.

But I am not very experienced with FiveAM, so probably I am wrong.

FiveAM doesn't play nice with ASDF

If you mean (asdf:test-system), it can be done by defining a test suite and test that suite.

[vindarel]

ACK.

It seems Prove is not archived anymore :p

[digikar99]

It seems Prove is not archived anymore :p

It isn't obsolete if it is going to be maintained, is it?

I think some of my issues with prove have more to do with my background - what would I need to study to understand the usage of pass, fail, skip, plan, finalize or to understand the ever-increasing number of tests when a test is run repeatedly:

(prove:deftest foo (is 5 5))
(prove:run-test 'foo) ; 1/1
(prove:run-test 'foo) ; 2/2
(prove:run-test 'foo) ; 3/3

May be I changed something, and I run the tests again - I'd expect the "test-numbers" would restart from 1, and not just continue. If something was broken before, and working now, this makes it not-so-easy to detect. Or may be, I'm missing the way it is supposed to be used.

What I did like about prove were the is- forms, and not seeing them elsewhere (did I overlook something?) spun up (yet another) test-lite. Even rove seems less on this. I'd be willing to fork prove (and may be maintain it) (or report and work with rove) but without the above doubts resolved, I can't seem to help. The state of testing-frameworks on cliki is already scary to say the least.

[bpseudopod]

Prove has been archived again.

[vindarel]

related #382 https://sabracrolleton.github.io/testing-framework

[vindarel]

(I'm in favor of FiveAM now too)

[digikar99]

Jeez, been a short while in the lisp world XD.

I've been using fiveam for its stability; unless I overlooked something, the comparison page has not yet benchmarked the compile and load-without-compile benchmarks of the frameworks. I felt this important in a use pattern where the tests and the code being tested are in the same system. And currently, fiveam fails miserably in this particular use case due to the use of eval - there is an open PR about this though.

[sheepduke]

By the way is FiveAM still maintained? The last commit was 13 months ago (not a big issue though) and there are open issues and PRs...

[sheepduke]

Submitted #404 for it.

[AlbertoEAF]

I used Rove more than a year ago, and it worked well. Also, it's maintained, with the latest commit this past July.

We should stick a notice that Prove was deprecated in favour of Rove until a rewrite for Rove is written.