Skip to content

worker: Overrule resource hog runner container on startup#3749

Merged
victorges merged 6 commits intomasterfrom
vg/fix/hungry-runner-restart
Sep 24, 2025
Merged

worker: Overrule resource hog runner container on startup#3749
victorges merged 6 commits intomasterfrom
vg/fix/hungry-runner-restart

Conversation

@victorges
Copy link
Copy Markdown
Contributor

@victorges victorges commented Sep 24, 2025

What does this pull request do? Explain your changes. (required)
This is to make sure that ai-runner containers that have gone rogue and are consuming
all of the system resources (normally VRAM) are able to get killed by a restarting O.

The problem used to be that the O died out of VRAM when starting and couldnt' get to kill
the runner. This change makes sure that we do it ASAP in the startup so that it frees resources
so the new O can start.

Specific updates (required)

  • Remove existing containers ASAP in worker startup

How did you test each of these updates (required)
Not yet.

Does this pull request close any open issues?
Fixes INF-350

Checklist:

@victorges victorges requested review from j0sh and mjh1 September 24, 2025 04:08
@linear
Copy link
Copy Markdown

linear bot commented Sep 24, 2025

@github-actions github-actions bot added go Pull requests that update Go code AI Issues and PR related to the AI-video branch. labels Sep 24, 2025
@victorges victorges changed the title worker/startup: Make sure resource hog runner container gets overruled worker/startup: Overrule resource hog runner container Sep 24, 2025
@victorges victorges changed the title worker/startup: Overrule resource hog runner container worker: Overrule resource hog runner container on startup Sep 24, 2025
@victorges victorges merged commit 825de00 into master Sep 24, 2025
4 of 7 checks passed
@victorges victorges deleted the vg/fix/hungry-runner-restart branch September 24, 2025 16:58
@codecov
Copy link
Copy Markdown

codecov bot commented Sep 24, 2025

Codecov Report

❌ Patch coverage is 12.12121% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 31.93740%. Comparing base (d8e520a) to head (41205d5).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
ai/worker/docker.go 19.04762% 14 Missing and 3 partials ⚠️
cmd/livepeer/starter/starter.go 0.00000% 11 Missing ⚠️
ai/worker/worker.go 0.00000% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##              master       #3749         +/-   ##
===================================================
- Coverage   31.94444%   31.93740%   -0.00704%     
===================================================
  Files            158         158                 
  Lines          47520       47543         +23     
===================================================
+ Hits           15180       15184          +4     
- Misses         31437       31455         +18     
- Partials         903         904          +1     
Files with missing lines Coverage Δ
ai/worker/worker.go 0.00000% <0.00000%> (ø)
cmd/livepeer/starter/starter.go 6.88863% <0.00000%> (-0.04378%) ⬇️
ai/worker/docker.go 69.15078% <19.04762%> (-1.61571%) ⬇️

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8e520a...41205d5. Read the comment docs.

Files with missing lines Coverage Δ
ai/worker/worker.go 0.00000% <0.00000%> (ø)
cmd/livepeer/starter/starter.go 6.88863% <0.00000%> (-0.04378%) ⬇️
ai/worker/docker.go 69.15078% <19.04762%> (-1.61571%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Issues and PR related to the AI-video branch. go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants