[BUG] Same test, different runs causes different heights with a weird compression effect

[filipesmedeiros]

Context:

• Playwright Version: 1.27.1
• Operating System: MacOS 13.0
• Node.js version: 16.15.0
• Browser: All
• Extra: Also CI using MacOS 12 (Github Actions)

Describe the bug

Subsequent screenshots (on different test runs) of the same page are coming out with different heights. This is mainly due to a weird "compression" (like smashing) effect on one of the screenshots.

These two example come from the same code, same server (NextJS locally) running, just yarn test two times in a row:

I'm sorry for the long image, but that may be part of the problem (although short pages sometimes have this problem too).

If you compare the images using something like https://www.diffchecker.com/image-diff/ you can see the weird effect.

Is this a known bug? The differences are even greater once we run the tests on CI (more screenshots are different), but it's flaky, so sometimes it happens and sometimes everything is good.

EDIT: Just to clarify, by compression I mean that the pixels themselves are smashed, so everything is shorter, which in turn causes the heights of the screenshots to differ, like on of them was "smashed", which is wrong, e.g. causes the square logo to turn into a rectangle, etc

Thanks in advance!

[pavelfeldman]

This looks like a bug to me. My only idea would be that the actual pixel data and the size of the page are captured in a racy manner and then the image is scaled based on the wrong size. Could you try waiting for layout to settle before you capture an image a little? Also, await expect.toHaveScreenshot should wait for the image to stabilize, are you using it?

[filipesmedeiros]

@pavelfeldman I used shouldMatchSnapshot on all runs, including the first. Is that bad? I know the docs say to use toHaveScreenshot but since shouldMatch also writes if not present, I assumed it was the same.

I can test with that, though! Thanks for the reply!

[filipesmedeiros]

Also, it's supposed to be expect(await page.screenshot).toMatchScreenshot right? We don't have to await expect like you said, I think (just checking)

[pavelfeldman]

We do, that's the point of resilient screenshotting, should be await expect(page).toHaveScreenshot().

[filipesmedeiros]

@pavelfeldman But expect doesn't return a Promise 😕

[filipesmedeiros]

Wait my bad, I'm not awaiting expect, but toHaveScreenshot. But toMatchSnapshot does not return a Promise

[filipesmedeiros]

Ok I see what's happening: I'm doing expect(await page.screenshot).toMatchSnapshot (like documented here https://playwright.dev/docs/test-assertions#screenshot-assertions-to-match-snapshot-1) but maybe that's wrong and I should be doing await expect(page).toHaveScreenshot. Let me try and I'll get back to you!

[filipesmedeiros]

Same thing happens 😞 e.g. in the above image you can see a 1 pixel height different between screenshots

[pavelfeldman]

That is something we might be able to fix!

[filipesmedeiros]

Ok I don't know if it helps, but running it inside Docker Linux (and CI on Linux) fixed it (or at least all the tests passed).

So the issue seems to be specific for Mac!

EDIT: nevermind, I think it was a flaky test that passed :(