Add TransferConfig support for CRTTransferManager #361
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for partial configuration of
CRTTransferManagerusingTransferConfig. It does the following:configparameter inCRTTransferManagerinitializer that gets passed intoS3ClientArgsCreator.S3ClientArgsCreatorlooks for supported CRT transfer config options and adds them to the request dict.UNSET_DEFAULTonTransferConfig. This value can be used to set defaults ontoTransferManagersubclasses and be able to disambiguate between a value that's there by default or a value that was explicitly provided. Details of its usage in boto3 can be found here: Add new preferred transfer client mode to enable CRT in all environments boto3#4629get_deep_attr()onTransferConfigthat can be used to access to real, underlying value of an attribute. This is useful in cases where__getattribute__has been overridden. Details of its usage in boto3 can be found here: Add new preferred transfer client mode to enable CRT in all environments boto3#4629TransferConfig.multipart_chunksizeisUNSET_DEFAULT(ie value was not explicitly provided), thenS3ClientArgsCreatorwill set{'part_size': None}and let CRT determine the part size.Although
UNSET_DEFAULTandget_deep_attr()are primarily used to support boto3's customTransferConfigclass, I decided to implement them in the base class in s3transfer becauseS3ClientArgsCreatorneeds to use them to determine whatpart_sizevalue to set. I wanted to avoid s3transfer classes from requiring knowledge about boto3 APIs since it doesn't depend on the boto3 library.