Skip to content

Conversation

fregataa
Copy link
Member

@fregataa fregataa commented Oct 24, 2024

resolves #2952

Add rate limiting for computing kernel creation. The number of kernels that can be created within a single scheduler tick is now capped by a shared config value.
The value is saved in shared config(etcd) and the key of config value is config/session/max-num-kernel-to-create. The default value is null, which does not limit the number of kernels created in a single tick.

Checklist: (if applicable)

  • Milestone metadata specifying the target backport version
  • API server-client counterparts (e.g., manager API -> client SDK)

@fregataa fregataa added this to the 24.03 milestone Oct 24, 2024
@fregataa fregataa self-assigned this Oct 24, 2024
@github-actions github-actions bot added comp:manager Related to Manager component size:M 30~100 LoC labels Oct 24, 2024
@fregataa fregataa requested a review from kyujin-cho October 24, 2024 08:41
@fregataa fregataa marked this pull request as ready for review October 24, 2024 08:42
@fregataa fregataa changed the title feat: limit max num of session to prepare in one tick feat: Add rate limiting for computing session creation Oct 24, 2024
@fregataa fregataa changed the title feat: Add rate limiting for computing session creation feat: Add rate limiting for computing kernel creation Oct 24, 2024
@fregataa fregataa requested a review from inureyes October 24, 2024 09:03
@fregataa fregataa added the action:on hold Hold it. Wait for the restart. label Oct 28, 2024
@fregataa
Copy link
Member Author

fregataa commented Oct 28, 2024

Added an "on-hold" label to this PR and converted this to a draft because we suspected that too few DB connections were causing this issue.

@fregataa fregataa marked this pull request as draft October 28, 2024 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
action:on hold Hold it. Wait for the restart. comp:manager Related to Manager component size:M 30~100 LoC
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Limit the number of sessions (and their kernels) created at a single scheduler tick
1 participant