Wilcoxon Signed Rank Test | Mann-Whitney-Wilcoxon Test - differences to R

[hkoestin]

Dear all,

recently, I decided to use the Accord.net framework, because it comes with implementations for different tests I need in one of my projects. Namely, these are the Wilcoxon Signed Rank Test for comparing a sample against a median value and the Mann-Whitney-Wilcoxon Test to compare two samples.

Now, as I started implementing things, I figured out, that the Accord.net results do not match the results coming from R.

Is there an explanation for the differences?
So far, I just found one hint about this, that R may calculate the ranks differently. See here: http://stats.stackexchange.com/questions/65844/wilcoxon-rank-sum-test-in-r

Thank you so much for any help concerning this topic!

See below my examples. Excuse the bad code, but it is just for demonstration purpose of the differences.

Cheers, Harald

---

C# Testing stuff

var s1 = new double[] { 35, 15, 25, 10, 45, 20, 21, 22, 30, 17 };
var s2 = new double[] { 20, 17, 23, 15, 49, 19, 24, 26, 33, 18 };

Console.WriteLine("C# Tests");

var mannWhitneyWilcoxonTest = new MannWhitneyWilcoxonTest(s1, s2, TwoSampleHypothesis.FirstValueIsGreaterThanSecond);
Console.WriteLine("MannWWT s1 > s2:  " + mannWhitneyWilcoxonTest.PValue + 
    " Stat1: " + mannWhitneyWilcoxonTest.Statistic1 + "; Stat2: " + mannWhitneyWilcoxonTest.Statistic2 +
    " - Significant: " + mannWhitneyWilcoxonTest.Significant + " - Hyp: " + mannWhitneyWilcoxonTest.Hypothesis);
mannWhitneyWilcoxonTest = new MannWhitneyWilcoxonTest(s1, s2, TwoSampleHypothesis.ValuesAreDifferent);
Console.WriteLine("MannWWT s1 <> s2: " + mannWhitneyWilcoxonTest.PValue +
    " Stat1: " + mannWhitneyWilcoxonTest.Statistic1 + "; Stat2: " + mannWhitneyWilcoxonTest.Statistic2 + 
    " - Significant: " + mannWhitneyWilcoxonTest.Significant + " - Hyp: " + mannWhitneyWilcoxonTest.Hypothesis);
mannWhitneyWilcoxonTest = new MannWhitneyWilcoxonTest(s1, s2, TwoSampleHypothesis.FirstValueIsSmallerThanSecond);
Console.WriteLine("MannWWT s1 < s2:  " + mannWhitneyWilcoxonTest.PValue +
    " Stat1: " + mannWhitneyWilcoxonTest.Statistic1 + "; Stat2: " + mannWhitneyWilcoxonTest.Statistic2 + 
    " - Significant: " + mannWhitneyWilcoxonTest.Significant + " - Hyp: " + mannWhitneyWilcoxonTest.Hypothesis);

var wilcoxonSignedRankTest = new WilcoxonSignedRankTest(s1, 13, OneSampleHypothesis.ValueIsGreaterThanHypothesis);
Console.WriteLine("WSRT > 13:  " + wilcoxonSignedRankTest.PValue +
    " Stat: " + wilcoxonSignedRankTest.Statistic +" - Significant: " +
    wilcoxonSignedRankTest.Significant + " - Hyp: " + wilcoxonSignedRankTest.Hypothesis);
wilcoxonSignedRankTest = new WilcoxonSignedRankTest(s1, 13, OneSampleHypothesis.ValueIsDifferentFromHypothesis);
Console.WriteLine("WSRT <> 13: " + wilcoxonSignedRankTest.PValue +
    " Stat: " + wilcoxonSignedRankTest.Statistic + " - Significant: " +
    wilcoxonSignedRankTest.Significant + " - Hyp: " + wilcoxonSignedRankTest.Hypothesis);
wilcoxonSignedRankTest = new WilcoxonSignedRankTest(s1, 13, OneSampleHypothesis.ValueIsSmallerThanHypothesis);
Console.WriteLine("WSRT < 13: " + wilcoxonSignedRankTest.PValue +
    " Stat: " + wilcoxonSignedRankTest.Statistic + " - Significant: " +
    wilcoxonSignedRankTest.Significant + " - Hyp: " + wilcoxonSignedRankTest.Hypothesis);

This produces

C# Tests
MannWWT s1 > s2:  0,507350234904414 Stat1: 50,5; Stat2: 49,5 - Significant: False 
MannWWT s1 <> s2: 0,985299530191171 Stat1: 50,5; Stat2: 49,5 - Significant: False 
MannWWT s1 < s2:  0,492649765095586 Stat1: 50,5; Stat2: 49,5 - Significant: False


WSRT > 13: 0,0029296875 Stat: 53 - Significant: True
WSRT <> 13: 0,005859375 Stat: 53 - Significant: True 
WSRT < 13:  0,9970703125 Stat: 53 - Significant: False 

And here the R code (and output) for testing:

a <- c(35, 15, 25, 10, 45, 20, 21, 22, 30, 17)
b <- c(20, 17, 23, 15, 49, 19, 24, 26, 33, 18)

wilcox.test(a,b, paired=FALSE, alternative="greater")     # W = 49.5, p-value = 0.5302  
wilcox.test(a,b, paired=FALSE, alternative="two.sided") # W = 49.5, p-value = 1 
wilcox.test(a,b, paired=FALSE, alternative="less")           # W = 49.5, p-value = 0.5 

wilcox.test(a, mu=13, alternative="greater")     # V = 53, p-value = 0.00293
wilcox.test(a, mu=13, alternative="two.sided") # V = 53, p-value = 0.005859
wilcox.test(a, mu=13, alternative="less")          # V = 53, p-value = 0.998

So the one-sample Wilcoxon Signed Rank Test produces quite similar results in R and C#.
But the Mann-W-W-Test not really. There, the difference is at about 3% some time. Why?

[cesarsouza]

Hi there,

Thanks for opening the issue! In fact, I have been researching this problem during the past week, and I found a couple issues with the current implementation of the Wilcoxon tests. I will be committing relevant fixes in the following days.

Many thanks again for such descriptive report, and for the invaluable examples.

Regards,
Cesar

[hkoestin]

Thank you @cesarsouza for taking up this issue.

It would be very crucial for us to have these tests (Wilcoxon Rank and Mann-Whitney-U) being implemented correctly and validated against R results, as researchers want to use the results within their papers and publications.

Are you planning to add certain kind of validation tests against R to the framework?

Regards,
Harald

[cesarsouza]

Hi Harald,

Yes, I understand. There are some individual tests to compare the results of some methods against those generated by R, but as it can be seen, the suite of tests is not complete and particularly didn't cover well the Wilcoxon hypothesis tests.

Any PR adding more tests to the framework, comparing the current results to R, would in fact be highly welcome.

I will commit the changes soon, and add some examples on how generate results comparable with R.

Regards,
Cesar

[cesarsouza]

Hi Harald,

I've just committed the fixes. I will generate a new pre-release NuGet package soon.

Examples reproducing the correct results for the values you included in your report can be seen in

• https://github.com/accord-net/framework/blob/development/Unit%20Tests/Accord.Tests.Statistics/Testing/WilcoxonSignedRankTestTest.cs#L226, and
• https://github.com/accord-net/framework/blob/development/Unit%20Tests/Accord.Tests.Statistics/Testing/MannWhitneyWilcoxonTestTest.cs#L396

I will update the examples in the documentation in the sequence.

Regards,
Cesar

[cesarsouza]

Fixed in 3.5.0.

[hkoestin]

Hey Cesar,

thanks for letting me know about the new version.

I just updated to 3.5.0 and can verify, that for the Wilcoxon-Signed-Rank-Test, results are much closer to R as they have been before.

However, for the Mann-Whitney-Wilcoxon Test I don't see much difference. It is still quite different than R outputs.

Having the following vectors in R, running tests using a 5% alpha value (= 0.95 confidence level), I get the following results:

a <- c(35,15,25,10,45,20,21,22,30,17)
b <- c(20,17,23,15,49,19,24,26,33,18)

c <- c(45,45,15,50,30,15,30,35,25)
d <- c(45,55,20,55,20,25,35,45,20)

wilcox.test(a,b, paired=FALSE, alternative="two.sided", conf.level=0.95, exact=TRUE) 
# --> W=49.5, p-value=1

wilcox.test(a,b, paired=FALSE, alternative="less", conf.level=0.95, exact=TRUE)
# --> W=49.5, p-value=0.5

wilcox.test(a,b, paired=FALSE, alternative="greater", conf.level=0.95, exact=TRUE) 
# --> W=49.5, p-value=0.5302


wilcox.test(c,d, paired=FALSE, alternative="two.sided", conf.level=0.95, exact=TRUE) 
# --> W=49.5, p-value=0.6557

wilcox.test(c,d, paired=FALSE, alternative="less", conf.level=0.95, exact=TRUE)
# --> W=49.5, p-value=0.3278

wilcox.test(c,d, paired=FALSE, alternative="greater", conf.level=0.95, exact=TRUE) 
# --> W=35, p-value=0.7037

When I input vectors a and b resp. c and d into the following test method of the framework, I get the following results: (alpha = 0.05)

var test = new MannWhitneyWilcoxonTest(sampleData1, sampleData2,
    TwoSampleHypothesis.ValuesAreDifferent, exact: true)
{
    Size = 0.05
};

// a,b --> p-Value = 0.985299
// c,d --> p-Value = 0.778815

So for vectors a and b, the accord p-value is quite close to the R value of 1, although still wrong by 1.5%.
For vectors c and d however, the accord p-value of 0.778815 is quite far from the R value of 0.6557. It is closer to the R value for the "greater" alternative, which is 0.7037.

Am I doing something wrong?
Am I right, that the TwoSampleHypothesis type of "ValuesAreDifferent" is the same as the R alternative "two.sided"?
R supports the parameters for "paired". What is the equivalent to this in Accord.Net? Is there a build-in assumption about the pairing of values?

Thanks for any input regarding this.

Regards,
Harald

[cesarsouza]

Hi Harald,

I am about to dive into the Wilcoxon tests again. Sorry for not having it all fixed at once.

I just wanted to confirm that you are right in your assumptions: "ValuesAreDifferent" should be equivalent to R's "two.sided", and MannWhitneyWilcoxonTest tests should be equivalent to R's tests with "paired = FALSE".

Also, note that even if you have specified "exact = TRUE" in the wilcox.test commands, R is not computing exact tests on these cases, assuming that you have got the same warning as me (Warning message: wilcox.test.default(c, d, paired = FALSE, alternative = "two.sided", : compute exact p-value with ties)

[cesarsouza]

Indeed, for the first example with vectors a, b this was actually the problem. In order to replicate R's results for those two vectors, you might have to specify exact: false when calling MannWhitneyWilcoxonTest's constructor:

    var mannWhitneyWilcoxonTest = new MannWhitneyWilcoxonTest(a, b,
                TwoSampleHypothesis.FirstValueIsGreaterThanSecond, exact: false);
    double p = mannWhitneyWilcoxonTest.PValue; // should be exactly 0.5302

Now, regarding the second pair of vectors, I suspect the problem might have something to do with the presence of ties in the data. I will continue trying to take a look.

[cesarsouza]

Indeed, the problem was that the function that identifies ties was not accounting properly for a tie that happens in the last elements. I will commit a fix soon.