Skip to content

Local mode fails to read serialized bytes from S3 alternative storage #5262

@cuongvd23

Description

@cuongvd23

Describe the bug
I encountered ServiceError: Failed to read serialized bytes from S3 alternative storage (MinIO).

To reproduce

  • Run MinIO image with docker, expose the server to port 9000, the endpoint address is http://localhost:9000.
  • Utilize LocalPipelineSession with s3_endpoint_url = 'http://localhost:9000'.
  • Start SageMaker local mode.

Expected behavior
Can read objects from MinIO without any problems.

Screenshots or logs

INFO [sagemaker.remote_function] Uploading serialized function code to s3://sagemaker-data/pipelines/step-name/function
INFO [sagemaker.remote_function] Uploading serialized function arguments to s3://sagemaker-data/pipelines/step-name/arguments
...
sagemaker.remote_function.errors.ServiceError: Failed to read serialized bytes from s3://sagemaker-data/pipelines/pipeline-name/step-name-xxx/xxx/function/metadata.json: ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden')

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.250.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): Not used
  • Framework version: Not used
  • Python version: 3.13.0
  • CPU or GPU: CPU
  • Custom Docker image (Y/N):

Additional context

  • Successfully upload the serialized resources (function code, function arguments) to MinIO.
  • Double-check on MinIO console to make sure the objects exist.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions