Bug Report: v18.0.0-rc1 S3 backup failed to upload

[L3o-pold]

Overview of the Issue

Upgraded from v17.0.3 to v18.0.0-rc1 our backup failed to be upload to S3 like storage (not AWS but more like Minio).

E1005 07:20:00.774018      14 main.go:56] rpc error: code = Unknown desc = TabletManager.Backup on commerce-0000001010 error: MultipartUpload: upload multipart failed
	upload id: ZGQ5OWUxNWYtY2IzNS00MDdjLWI1OGItMWNkZDA4M2ZlYzRk
caused by: RequestCanceled: request context canceled
caused by: context canceled: MultipartUpload: upload multipart failed
	upload id: ZGQ5OWUxNWYtY2IzNS00MDdjLWI1OGItMWNkZDA4M2ZlYzRk
caused by: RequestCanceled: request context canceled
caused by: context canceled

Reproduction Steps

/vt/bin/vtctldclient --server=vtctld:15999 --logtostderr=true BackupShard --concurrency=1 "commerce/-"

Running vttablet and vtctld with following flag:

--backup_storage_implementation=s3 \
--backup_engine_implementation=xtrabackup \
--s3_backup_aws_region=us-east-1 \
--s3_backup_storage_bucket=default \
--s3_backup_force_path_style=true \
--s3_backup_aws_endpoint=https://* \
--s3_backup_storage_root=*

Binary Version

vtctldclient version Version: 18.0.0-rc1 (Git revision 6ab165ade925b35a00cf447827d874eba13998b6 branch 'heads/v18.0.0-rc1') built on Tue Oct  3 15:00:58 UTC 2023 by vitess@buildkitsandbox using go1.21.1 linux/amd64

Operating System and Environment details

kubernetes / docker official vitess image

Log Fragments

No response

[deepthi]

@L3o-pold can you please capture the log from vttablet as well? None of us maintainers have access to a minio environment so this will be tough to debug.

[deepthi]

Our minio dependency hasn't changed in years. However the aws-sdk-go dependency has changed. That would be the first place to check.
https://github.com/vitessio/vitess/blame/main/go.mod#L14
You can see the diff here 9d2029a

If you are able to build an image with the older version of aws-sdk-go, and that works, that will give us a starting point. But in any case we'll need the vttablet logs, not just the vtctld logs.

[L3o-pold]

@deepthi we are not using Minio but AWS compatible storage like it.
vttablet logs are the same:

vttablet W1006 00:03:11.814578       1 rpc_server.go:72] TabletManager.Backup(concurrency:1)(on commerce-0000001012 from ) error: MultipartUpload: upload multipart failed
vttablet 	upload id: M2MzYjJkY2YtODI4Yi00NGE5LWFkMjgtYTIwYzU2NmM4Yjdh
vttablet caused by: RequestCanceled: request context canceled
vttablet caused by: context canceled

I'll try to see if I can reproduce with a simple Minio docker instance, and with the team in charge of our storage solution to see if they have some log on their side.

[L3o-pold]

I reproduce with a Minio docker instance:

    minio:
        image: minio/minio:RELEASE.2023-09-30T07-02-29Z
        command: ["server", "/data", "--console-address", ":9001"]
        environment:
            MINIO_ROOT_USER: "test"
            MINIO_ROOT_PASSWORD: "test"

[L3o-pold]

From my testing, downgrading the aws-sdk-go have no impact, but reverting #12500 works.

[deepthi]

I'm going to test with vanilla S3 to see if that exhibits the same problem. If yes, we will have to revert at least a portion of #12500 and then figure out a way forward.

[deepthi]

I have tested PR #12500 (or actually it's exact counterpart on the planetscale fork) and backups with vanilla S3 work just fine. However, my test db is very small, and the whole vtbackup process (download previous backup from s3, restore and take a new backup and upload to s3) takes 10 minutes.
How big is your backup? Context canceled is telling us that a timeout is being hit.
Default timeout (--action_timeout) for all vtctl commands is 1 hour. Does your command take longer than that? Or does it error out exactly after 1 hour?
If yes, you need to extend that. If not, let me know and we'll see if we can think of something else.

[L3o-pold]

@deepthi 2GB and fails in less than 2ms once xtrabackup finished.

edit: tried with an empty database and same issue.

[harshit-gangal]

@deepthi Is planetscale using the new vtctld grpc api to do the backup?

[deepthi]

@L3o-pold can you try if builtinbackup works? You did specify the flags in your issue description but I missed the fact that you are in fact using xtrabackup. My test was with vtbackup (which operator uses) + builtin backup, not xtrabackup.

Also:

• were you using the same command with v17? Or were you by any chance using vtctlclient instead of vtctldclient?

[L3o-pold]

@deepthi --backup_engine_implementation=xtrabackup was specified in the issue yeah :) I'll try with builtin backup