drivers: Refactor package as a more generic and performant library#2223
Merged
drivers: Refactor package as a more generic and performant library#2223
Conversation
5da8ff8 to
ad83025
Compare
ad83025 to
32501a0
Compare
a33e9b4 to
9f8c762
Compare
leszko
approved these changes
Mar 30, 2022
Contributor
leszko
left a comment
There was a problem hiding this comment.
Added 2 comments, other than that LGTM 👍
Also remember to:
- Update CHANGELOG_PENDING.md
- Consider squashing some commits before merging
9f8c762 to
8726c58
Compare
8726c58 to
f142c67
Compare
The Filebase S3-compatible API does not return a file size in the response for files greater than 255 bytes. That caused a panic on s3.go:267 when reading recordings. With this change, the problem won't happen anymore cause we don't even try to dereference the ContentLength, but hold it as a nilable pointer that should be checked wherever it is used (right now: nowhere).
We want to use this lib for saving large files to ObjectStore as well for the VOD tasks. Since they can be really large, we don't want to have to buffer the whole files in memory for saving. So changing this API to receive a Reader instead makes more sense.
Avoid the need to buffer the whole input in memory before saving. Will also be more performant for large files.
Increase the concurrency and the part size so the uploads are more performant in our production environment. These are also more compatible with Storj's configuration which saves files in 64MB chunks on the decentralized network, so that part size also works well for them. But it still improves upload performance to gcloud in around 10x as well.
f142c67 to
b76b14f
Compare
Codecov Report
@@ Coverage Diff @@
## master #2223 +/- ##
===================================================
- Coverage 54.93891% 54.87805% -0.06086%
===================================================
Files 93 93
Lines 19316 19352 +36
===================================================
+ Hits 10612 10620 +8
- Misses 8115 8140 +25
- Partials 589 592 +3
Continue to review full report at Codecov.
|
It returns an error so had to change a few signatures.
Also add comments with context about how those numbers came up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this pull request do? Explain your changes. (required)
This is to make some improvements to the
driverspackage so that it can also be usedas a library from other projects, specifically the
task-runnerfor VOD tasks. It is currentlyusing the drivers library for a platform-agnostic way of accessing Object Stores for reading
and writing VOD assets.
The greatest advantage of that is that we can just use the same Object Store API from
livepeer-comto declare where assets should be stored as well as stream recordings,keeping the whole interface leaner.
Specific updates (required)
Sizeproperty optional in ourFilerepresentation. It is optional in the AWS SDK and is not returned in all implementations of it, like previous versions of Filebase, so it was required at some point (not anymore) but I still think it makes sense to keep it.io.Readerinstead of the whole buffered file. The reader is compatible with clients that have the whole bytes in memory, but requiring all clients to buffer the entire contents in memory is not ideal. This would have made the VOD tasks much more inefficient, needing to read the entire files in memory before sending to the OS.io.ReadSeekerdue to hashing purposes. The uploader is the suggested solution in the open issue in their SDK repo.How did you test each of these updates (required)
go-livepeerdriverslib in thetask-runnerfor VOD and already deployed it in production, it's working greatDoes this pull request close any open issues?
Part of livepeer/task-runner#14
Checklist:
makeruns successfully./test.shpass(does it need a changelog update?)