Skip to content

chore: Bypass Firefox for Playwright test due to flakiness#8050

Merged
rjsparks merged 1 commit intoietf-tools:mainfrom
holloway:bypass-firefox-playwright
Oct 15, 2024
Merged

chore: Bypass Firefox for Playwright test due to flakiness#8050
rjsparks merged 1 commit intoietf-tools:mainfrom
holloway:bypass-firefox-playwright

Conversation

@holloway
Copy link
Contributor

These tests don't test anything Firefox-specific (it's conventional JavaScript mostly dependant on a 3rd-party library) so we've decided to bypass these tests in Firefox and just rely on Chrome.

@codecov
Copy link

codecov bot commented Oct 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.89%. Comparing base (c7f6bde) to head (59bc498).
Report is 96 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8050      +/-   ##
==========================================
+ Coverage   88.78%   88.89%   +0.10%     
==========================================
  Files         296      304       +8     
  Lines       41320    41302      -18     
==========================================
+ Hits        36687    36715      +28     
+ Misses       4633     4587      -46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@holloway
Copy link
Contributor Author

fyi @rjsparks

Copy link
Member

@jennifer-richards jennifer-richards left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to simply remove Firefox from the list being tested, or is skipping it the only way to do that?

I think part of our motivation for testing against firefox was to ensure we're not using corners that are browser-dependent. Not for this PR, but is there some alternative way to test that?

@rjsparks
Copy link
Member

I think we had also hoped that testing against Firefox and Chrome would also give us an early warning if either browser changed behavior in a way it shouldn't that we would need to adjust for (this has happened to us at several meetings).

I agree that right now the firefox playwright tests are not helping us because of the random inexplicable failures. I'm ok with muting that noise for the moment, but I think we should put pressure on making sure we're not surprised by activity at firefox.

@NGPixel
Copy link
Member

NGPixel commented Oct 15, 2024

One reason to keep the Firefox tests is to make sure we don't start using a CSS or JS feature that's not yet supported in Firefox. Firefox tends to be behind times when it comes to implementing new standards.

@holloway
Copy link
Contributor Author

My understanding is that only the status.spec.js test is flakey in Firefox, so I've tried to narrowly disable only that.

@rjsparks rjsparks merged commit 68ceea9 into ietf-tools:main Oct 15, 2024
@rjsparks
Copy link
Member

merged, but I have to wonder if it was telling us there's a problem with the status notifications on Firefox?

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants