Attempt Deflake - Disable JS in FileParameterValueTest and DirectoryBrowserSupportTest#10471
Conversation
…archy Test failure The test `hudson.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy` failed in CI with error, <details><summary>failure log</summary> ``` java.lang.NullPointerException: Cannot invoke "org.htmlunit.html.DomElement.getHtmlElementDescendants()" because "doc" is null at org.htmlunit.html.HtmlPage.executeDeferredScriptsIfNeeded(HtmlPage.java:1466) at org.htmlunit.html.HtmlPage.initialize(HtmlPage.java:265) at org.htmlunit.WebClient.loadWebResponseInto(WebClient.java:701) at org.htmlunit.WebClient.loadWebResponseInto(WebClient.java:575) at org.htmlunit.WebClient.getPage(WebClient.java:493) at org.htmlunit.WebClient.getPage(WebClient.java:402) at org.htmlunit.WebClient.getPage(WebClient.java:538) at org.htmlunit.WebClient.getPage(WebClient.java:520) at org.jvnet.hudson.test.JenkinsRule$WebClient.getPage(JenkinsRule.java:2740) at org.jvnet.hudson.test.JenkinsRule.after(JenkinsRule.java:519) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:677) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.lang.Thread.run(Thread.java:840) ``` </details> This is noticed in couple of PRs failing currently, * jenkinsci#10456 ([ref failed test result](https://ci.jenkins.io/job/Core/job/jenkins/job/PR-10456/7/testReport/)) * jenkinsci#10467 ([ref failed test result](https://ci.jenkins.io/job/Core/job/jenkins/job/PR-10467/2/testReport/)) I haven't been able to reproduce in local, but theory is these below mentioned delays could be the cause, * there is a slowness noticed in calls `wc.goTo(p.getUrl()...)` taking `~10s`. * And once the test is complete the stopping is taking another `10s`. <details><summary>log</summary> ``` [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running hudson.model.FileParameterValueTest === Starting fileParameter_canStillUse_internalHierarchy(hudson.model.FileParameterValueTest) 0.047 [id=21] INFO o.jvnet.hudson.test.WarExploder#explode: Using jenkins.war resources from /.../jenkins/war/target/jenkins 0.304 [id=21] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer2: Running on http://localhost:64862/jenkins/ 0.367 [id=21] INFO jenkins.model.Jenkins#<init>: Starting version 2.504-SNAPSHOT 0.421 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 1.002 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/command-launcher.jpi as a dependency of gradle 1.012 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/apache-httpcomponents-client-4-api.jpi as a dependency of gradle 1.034 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/jdk-tool.jpi as a dependency of gradle 1.052 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/trilead-api.jpi as a dependency of gradle 1.079 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/eddsa-api.jpi as a dependency of gradle 1.084 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/gson-api.jpi as a dependency of gradle 1.098 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/mina-sshd-api-common.jpi as a dependency of gradle 1.109 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/mina-sshd-api-core.jpi as a dependency of gradle 1.134 [id=46] INFO hudson.PluginManager#considerDetachedPlugin: Loading a detached plugin /var/folders/80/w8mmdy513sb3y9fp5fr7vvqc0000gn/T/jenkins1642946911825735225/sshd.jpi as a dependency of gradle 1.353 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 1.362 [id=37] INFO j.b.api.BouncyCastlePlugin#start: /.../jenkins/test/target/j h15288515834251035408/plugins/bouncycastle-api/WEB-INF/optional-lib not found; for non RealJenkinsRule this is fine and can be ignored. 1.931 [id=53] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 1.937 [id=34] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 1.937 [id=47] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 2.254 [id=53] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded 2.255 [id=53] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted 2.255 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 2.255 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated 2.307 [id=50] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization 2.466 [test0 jenkinsci#1] Started by user SYSTEM 2.466 [test0 jenkinsci#1] Running as SYSTEM 2.466 [test0 jenkinsci#1] Building in workspace /.../jenkins/test/target/j h15288515834251035408/workspace/test0 2.466 [test0 jenkinsci#1] Copying file to direct-child1.txt 2.466 [test0 jenkinsci#1] Copying file to parent/child2.txt 2.899 [test0 jenkinsci#1] Finished: SUCCESS 3.046 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: getting ws 14.449 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: got htmlunit for ws 14.450 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: got ws 14.452 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: getting ws/parent 24.720 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: got htmlunit for ws/parent 24.733 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: got ws/parent 24.733 [id=21] INFO h.model.FileParameterValueTest#fileParameter_canStillUse_internalHierarchy: completed 34.767 [id=21] INFO hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. WARN: Please see http://www.slf4j.org/codes.html#release for an explanation. 34.799 [id=21] INFO hudson.lifecycle.Lifecycle#onStatusUpdate: Jenkins stopped 34.840 [id=21] INFO o.j.h.t.TemporaryDirectoryAllocator#dispose: deleting /.../jenkins/test/target/j h15288515834251035408 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.15 s -- in hudson.model.FileParameterValueTest ``` </details>
|
Checking the Edit: now filed jenkinsci/jenkins-test-harness#946 |
df441f2 to
0a394ea
Compare
jglick
left a comment
There was a problem hiding this comment.
Did not look at details at all, but it is commonplace to need to disable JS in WebClient to make tests run more quickly or reliably, assuming of course that the test assertions are not specifically checking something dynamic in the page.
…SupportTest These two tests looked like frequently failing in the test paths that visit a builds `ws` page, which is having the javascript execution issue jenkinsci/jenkins-test-harness#946 Here trying to disable the javascript in all the webclient usages in these two tests. Note, I have disabled in the test methods that didn't visit the `ws` page specifially to keep the `getWebClient()` usage uniform.
| JenkinsRule.WebClient wc = j.createWebClient(); | ||
| JenkinsRule.WebClient wc = getWebClient(); | ||
| wc.getOptions().setThrowExceptionOnFailingStatusCode(false); | ||
| } |
There was a problem hiding this comment.
(seems like it was missed to check the intent // ensure also the file is not reachable by request (from commit)- I will file a separate PR for completing this test, should be couple of lines only)
| JenkinsRule.WebClient wc = j.createWebClient(); | ||
| JenkinsRule.WebClient wc = getWebClient(); | ||
| wc.getOptions().setThrowExceptionOnFailingStatusCode(false); | ||
| } |
timja
left a comment
There was a problem hiding this comment.
Looks ok in this case I would be wary of doing this too much as users will be running with JavaScript on in almost all cases and what if the experience breaks in that case.
I've checked the tests here and it seems fine for this though
|
/label ready-for-merge This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback. |
This PR proposes to disable the javascript in
WebClientin all places in the mentioned two test classes,wsURL.getWebClient()is now called in several methods, to keep consistency with the entire class - using it all methods.Besides possibly it fixing the flakiness, it speeds up the tests (slowness affected by jenkinsci/jenkins-test-harness#946),
01:29 min23.766s02:49 min37.973 sNoticed that two tests failed in CI with same kind of error,
FileParameterValueTest#fileParameter_canStillUse_internalHierarchyDirectoryBrowserSupportTest.tmpNotListedfailure log
Example CI Failure reports
FileParameterValueTest#fileParameter_canStillUse_internalHierarchyFileParameterValueTest#fileParameter_canStillUse_internalHierarchyDirectoryBrowserSupportTest.tmpNotListedI didn't reproduce the error, but hopefully it fixes the flakiness, possibly causes by the javascript issue.
example showing slowness
Testing done
Tests run in local, and faster
Before: execution summary log
After: execution summary log
Proposed changelog entries
Proposed changelog category
/label skip-changelog,tests
Proposed upgrade guidelines
N/A
Submitter checklist
@Restrictedor have@since TODOJavadocs, as appropriate.@Deprecated(since = "TODO")or@Deprecated(forRemoval = true, since = "TODO"), if applicable.evalto ease future introduction of Content Security Policy (CSP) directives (see documentation).Desired reviewers
@mention
Before the changes are marked as
ready-for-merge:Maintainer checklist
upgrade-guide-neededlabel is set and there is a Proposed upgrade guidelines section in the pull request title (see example).lts-candidateto be considered (see query).