Skip to content

Conversation

@romanisb
Copy link
Contributor

This contribution proposes a new PipelineRunOptions which provides an option to set a specific timeout within the runner is periodically trying to initialize the log output stream.

This option will replace the hardcoded retrycount with a more dynamical approach.

Without increasing the duration of this retry, we occasionally ran into issues as shown below. Especially in a Kubernetes environment in which the nodes is under high utilization.

Cloning into '.'...
Checking out pipeline from revision master
Your branch is up to date with 'origin/master'.
Already on 'master'
Delete pipeline git clone credentials
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/app/repo/org/jvnet/hudson/xstream/1.4.7-jenkins-1/xstream-1.4.7-jenkins-1.jar) to field java.util.TreeMap.comparator
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fields
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-01-15 07:50:03.002+0000 [id=1]	SEVERE	h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (main/1) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.io.FileNotFoundException: /jenkins_home/jobs/job/builds/1/log (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
	at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:399)
	at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:121)
	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:212)
	at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:165)
	at hudson.model.Run.writeWholeLogTo(Run.java:1556)
	at io.jenkins.jenkinsfile.runner.Runner.writeLogTo(Runner.java:134)
	at io.jenkins.jenkinsfile.runner.Runner.run(Runner.java:94)
Caused: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.jenkins.jenkinsfile.runner.JenkinsfileRunnerLauncher.doLaunch(JenkinsfileRunnerLauncher.java:32)
	at io.jenkins.jenkinsfile.runner.JenkinsLauncher.launch(JenkinsLauncher.java:83)
	at io.jenkins.jenkinsfile.runner.App.run(App.java:13)
	at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.run(Bootstrap.java:312)
	at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:135)

Testing done

### Submitter checklist
- [x] Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!
- [x] Ensure that the pull request title represents the desired changelog entry
- [x]  Please describe what you did
- [ ] Link to relevant issues in GitHub or Jira
- [ ] Link to relevant pull requests, esp. upstream and downstream changes
- [ ] Ensure you have provided tests - that demonstrates feature works or fixes the issue

romanisb and others added 6 commits June 28, 2023 13:20
This option allows the consumer to define how often the write loop
will retry writing to the provided output stream instead of having
a hardcoded default which in some cases is not sufficient enough.
@romanisb romanisb requested a review from a team as a code owner June 30, 2023 07:14
@romanisb romanisb changed the title Args/retry log write Add --write-log-init-timeout option to allow specifying a timeout to retry writing logs to stdout Jun 30, 2023
Copy link
Member

@oleg-nenashev oleg-nenashev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! It is a good patch

@oleg-nenashev oleg-nenashev added the enhancement New feature or request label Aug 6, 2023
@oleg-nenashev oleg-nenashev merged commit 58b0a62 into jenkinsci:main Aug 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants