Test are failing locally

[thomaspink]

Bug, feature request, or proposal:

Bug

What is the expected behavior?

Tests are successful on local machine

What is the current behavior?

Some tests are failing because of strange float/int value comparisons or wrong comma for fractional digits.

E.g.:

Chrome 55.0.2883 (Mac OS X 10.12.2) MdRipple basic ripple sizes ripple to cover element FAILED
Expected -229 to be close to -229.50849718747372, 1.
Expected -204 to be close to -204.50849718747372, 1.
Expected NaN to be close to 559.0169943749474, 1.
...

or

Chrome 55.0.2883 (Mac OS X 10.12.2) MdSlider slider with value property binding should update when bound value changes FAILED
Expected 'translateX(0px) scaleX(0,5)' to contain 'scaleX(0.5)'.
Expected 'translateX(0px) scaleX(0,75)' to contain 'scaleX(0.75)'.
...

Here is the complete log as a gist:
https://gist.github.com/thomaspink/87f6cacc047eca03dcbdbf1df926455c

What are the steps to reproduce?

Checkout master, install deps, run tests. Maybe it is only on my machine.

What is the use-case or motivation for changing an existing behavior?

Trying to build test cases for some PRs.
Also happens with clean master checkout.

Which versions of Angular, Material, OS, browsers are affected?

Version: master
Browser: Chrome 55.0.2883
OS: (Mac OS X 10.12.2)
Node Version: v6.9.4 (same with 7.0 or 5.10.1)

Is there anything else we should know?

Travis test work fine. Locally they throw an error. Tried switching node versions, reinstalling npm packages, ...

@jelbourn @devversion maybe you guys can help me out :/

[thomaspink]

Instead of expect(getStyle(tiles[1], 'width')).toBe('99.25px');, maybe we should parse the values in to numbers and then compare them?

[devversion]

@thomaspink This happens a lot on my machine as well. Those are mostly pixel deviations of inaccurate tests. Those tests should be updated to work better on other browsers.

I started working on making some tests (that fail on my machine) more stable, so hopefully I will be able to push a PR soon.

[thomaspink]

@devversion if you need help, i can assist you with updating the cases.
It's blocking me to push valid test for PRs, ... so i think is should be high prio

Short summary:

• MdRipple (floating point issues, NaN issues)
• MdGridList (decimal point errors)
• MdSlider (decimal point errors)
• MdTabGroup (async timeout errors)
• MdAutocomplete (async timeout errors)

btw. Are you even sleeping sometimes ;)
You are always responding within minutes :) big thumbs up 👍

[crisbeto]

I think that these are kinda random. On my machine I have a couple of similar failures in the MdSelect, but none in the above-mentioned components.

[devversion]

@thomaspink I shouldn't block you on fixing those. What did you figure out about those tests yet?

And yeah I do sleep 😄

[jelbourn]

Pretty odd- I never see these failures and the CI hasn't reported them. My workstation and Travis are Linux, and Kara works on her Mac and hasn't said anything about seeing these.

[devversion]

@jelbourn Kristiyan and I are on Windows and (from what I heard) Kristiyan also experiences pixel deviations randomly. Likely ~2 tests always fail locally for me.

[crisbeto]

The tests actually ran fine up until Chrome 54 or 55, at which point they messed with the zoom level.

[devversion]

It looks like the height of the Chrome browser is for some reason very uneven.

It looks like this happens on Windows.. looking why the height is not even.

[thomaspink]

Update from my side:
Did some debugging on the sizes ripple to cover element test which is the first test that is failing for me.

The first thing i stumbled upon was that the style properties (e.g. ripple.style.left) are returning wrong values. E.g.: -229,508px for the left property. You can see it has a wrong floating point char (, instead of .). This leads on the one hand to failing comparisons in the test and on the other hand to a wrong parsing to float (-229,508 parses to -229 instead of -229.508). So the test where toBeCloseTo is used are also failing.

Added replace(',', '.') to the pxStringToFloat function on my WIP PR, so these Tests are now green.

/** Extracts the numeric value of a pixel size string like '123px'.  */
const pxStringToFloat = (s: string) => {
  return parseFloat(s.replace('px', '').replace(',', '.'));
};

The second problem i got was that ripple.style.height has no value so if you parse it to float you will get NaN. Also ripple.clientHeight is 0. I don't get why this is happening. If i look at them in the demo app, the height is set :/

[thomaspink]

Update:

Tried to find out why float values have the wrong separator. Found this, so i tried it myself:

At first i ran document.body.style.left = "3.2323%" in chrome on the demo app and other websites. When you take a look on the first screen, the value is correct. Then i did the same in the chrome window which opens when you start the tests. There the value is 3,2323%. So it seems to be a chrome bug?

Demo App (normal Chrome, Version 55.0.2883.95 (64-bit)):

Karma Tests (Chrome opened by console, Version 55.0.2883.95 (64-bit)):

[thomaspink]

Got it - for some reason these floating point errors occur if the language in Chrome 55 on MacOS is German (maybe also other languages). I switched my OS to English and voilà - everything is fine.
The only thing is that i sometimes get a timeout error on MdMenu (Only in 1 out 4 runs).

MdMenu should close the menu when a click occurs outside the menu FAILED
Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

So as a conclusion: There is, at least for me, no bug in the tests/build in material2. It seems to be an issue in Chrome on MacOS when using it in German.

I think if we can solve the errors @devversion and @crisbeto are having, the issue can be closed.

[devversion]

@thomaspink Thanks for the investigations you did. The issues @crisbeto and I are experiencing on Windows are quite different than the ones you are explaining here.

I really start thinking that Windows does something weird with the scaling of Chrome.

I can fix my errors by resizing the Chrome window to 100% height (but not using the maximize icon - because that chanegs the height of the program-bar)

It's hard to explain what issues I experience here - I started playing with some Chromium flags to fix these issues, but didn't find a proper one yet.