vulkan upload weights per layer, reduce peak ram usage - #6534
Conversation
|
|
There was a problem hiding this comment.
Pull request overview
This pull request refactors Vulkan model weight uploading to reduce peak RAM usage by uploading weights layer-by-layer instead of all at once. The key optimization is chunking uploads into 256MB batches, which reduces memory pressure during model loading.
Changes:
- Removed centralized
NetPrivate::upload_model()method and moved upload logic inline intoNet::load_model()to enable per-layer uploads - Added
VkTransfer::reset()andVkTransfer::pending_upload_total()methods to support incremental uploads with periodic submission - Implemented upload chunking logic that submits and resets the VkTransfer command buffer after accumulating 256MB of pending uploads
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/net.cpp | Moved weight upload logic inline with model loading loop; added 256MB chunking to reduce peak memory usage |
| src/command.h | Added reset() and pending_upload_total() methods to VkTransfer class API |
| src/command.cpp | Implemented upload tracking, reset functionality, and getter for pending upload size |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6534 +/- ##
==========================================
- Coverage 92.96% 92.94% -0.02%
==========================================
Files 809 809
Lines 256977 256643 -334
==========================================
- Hits 238900 238546 -354
- Misses 18077 18097 +20 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
No description provided.