chore: Bypass Firefox for Playwright test due to flakiness#8050
chore: Bypass Firefox for Playwright test due to flakiness#8050rjsparks merged 1 commit intoietf-tools:mainfrom
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #8050 +/- ##
==========================================
+ Coverage 88.78% 88.89% +0.10%
==========================================
Files 296 304 +8
Lines 41320 41302 -18
==========================================
+ Hits 36687 36715 +28
+ Misses 4633 4587 -46 ☔ View full report in Codecov by Sentry. |
|
fyi @rjsparks |
jennifer-richards
left a comment
There was a problem hiding this comment.
Is there a way to simply remove Firefox from the list being tested, or is skipping it the only way to do that?
I think part of our motivation for testing against firefox was to ensure we're not using corners that are browser-dependent. Not for this PR, but is there some alternative way to test that?
|
I think we had also hoped that testing against Firefox and Chrome would also give us an early warning if either browser changed behavior in a way it shouldn't that we would need to adjust for (this has happened to us at several meetings). I agree that right now the firefox playwright tests are not helping us because of the random inexplicable failures. I'm ok with muting that noise for the moment, but I think we should put pressure on making sure we're not surprised by activity at firefox. |
|
One reason to keep the Firefox tests is to make sure we don't start using a CSS or JS feature that's not yet supported in Firefox. Firefox tends to be behind times when it comes to implementing new standards. |
|
My understanding is that only the |
|
merged, but I have to wonder if it was telling us there's a problem with the status notifications on Firefox? |
These tests don't test anything Firefox-specific (it's conventional JavaScript mostly dependant on a 3rd-party library) so we've decided to bypass these tests in Firefox and just rely on Chrome.