Latest release causes stdout issues - stdout hanging on multiple commands

[denizsokullu]

NOTE

If this case is urgent, please subscribe to Subnet so that our 24/7 support team may help you faster.

Expected Behavior

The command should run and finish. Edit: I guess this is a mc issue, not minio.

Current Behavior

The command hangs in CI(linux environment)

#!/bin/bash -eo pipefail
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc config host add minio $MINIO_ENDPOINT $MINIO_KEY $MINIO_SECRET
./mc mb minio/bucket1
./mc mb minio/bucket2

Output:

--2022-01-25 23:47:29--  https://dl.min.io/client/mc/release/linux-amd64/mc
Resolving dl.min.io (dl.min.io)... 178.128.69.202, 138.68.11.125
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22315008 (21M) [application/octet-stream]
Saving to: ‘mc’

mc                  100%[===================>]  21.28M  15.2MB/s    in 1.4s    

2022-01-25 23:47:31 (15.2 MB/s) - ‘mc’ saved [22315008/22315008]

�]11;?�\mc: Configuration written to `/home/circleci/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/home/circleci/.mc/share`.
mc: Initialized share uploads `/home/circleci/.mc/share/uploads.json` file.
mc: Initialized share downloads `/home/circleci/.mc/share/downloads.json` file.
Added `minio` successfully.
�]11;?�\Bucket created successfully `minio/bucket1`.
�]11;?�\

Possible Solution

Splitting the commands right after ./mc config ... fixes it.

Steps to Reproduce (for bugs)

1. The commands below are ran in a linux based container on CircleCI(a CI provider)

#!/bin/bash -eo pipefail
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc config host add minio $MINIO_ENDPOINT $MINIO_KEY $MINIO_SECRET
./mc mb minio/bucket1
./mc mb minio/bucket2

3. It hangs after

Context

This is a pipeline that has been running fine for 2 years-ish. We haven't updated any of the base images or configuration. Minio released a new version yesterday and it exactly started happening after that release. I'm not saying the release is the cause but I don't have much else to point at.

One note. The symbols that can't be shown properly in the stdout did not happen on the release before the latest. It might be indicating a problem.

Regression

YES

Latest release

Your Environment

• Version used (minio --version): Latest release
• Server setup and configuration: docker based CI with https://hub.docker.com/_/php using amd64 arch
• Operating System and version (uname -a): the above

[vadmeste]

Minio released a new version yesterday and it exactly started happening after that release

I guess you need to look for logs there

[denizsokullu]

I don't see anything suspicious in https://github.com/minio/mc/releases/tag/RELEASE.2022-01-25T21-02-01Z except for a PR that updated the loading spinners for the tool. Any thoughts on how that could be affecting this? If you don't think this is minio/mc related you can close this issue.

[SethTales]

I am also experiencing this same problem

[vadmeste]

Can we articulate what is exactly the problem ?

[denizsokullu]

I've put together this Dockerfile and small script but I can't seem to recreate the issue on it. It's as close as I can get to the environment.(docker build .; docker run -i -t {the sha of the image})

Dockerfile

FROM circleci/php:7.3
COPY minio.sh minio.sh
CMD ./minio.sh

minio.sh

#!/bin/bash

sudo wget https://dl.min.io/server/minio/release/linux-amd64/minio
sudo chmod +x minio
sudo mkdir ./data
sudo chown -R circleci ./data
sudo chmod u+rxw ./data
./minio server ./data --address=:3000 &

sleep 2

sudo wget https://dl.min.io/client/mc/release/linux-amd64/mc
sudo chmod +x mc

./mc config host add minio http://127.0.0.1:3000 minioadmin minioadmin
./mc mb minio/test1
./mc mb minio/test2

There might be something that I don't know about the CI environment but here is the 'definition of the issue':
Any 3 mc commands run one after another cause the last one to hang and never finish. I have done something to alleviate the issue and have no more time to investigate the reason behind it.

[harshavardhana]

./mc config host add minio http://127.0.0.1:3000 minioadmin minioadmin
./mc mb minio/test1
./mc mb minio/test2

Can you add --debug for mc

[denizsokullu]

#!/bin/bash -eo pipefail
./mc mb --debug minio/test1
./mc mb --debug minio/test2
./mc mb --debug minio/test3

�]11;?�\mc: <DEBUG> PUT /test1/ HTTP/1.1
Host: 127.0.0.1:3000
User-Agent: MinIO (linux; amd64) minio-go/v7.0.20 mc/RELEASE.2022-01-25T21-02-01Z
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20220126/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20220126T194340Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Wed, 26 Jan 2022 19:43:40 GMT
Location: /test1
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 16CDE98A854E77B8
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  1.315734ms

Bucket created successfully `minio/test1`.
�]11;?�\mc: <DEBUG> PUT /test2/ HTTP/1.1
Host: 127.0.0.1:3000
User-Agent: MinIO (linux; amd64) minio-go/v7.0.20 mc/RELEASE.2022-01-25T21-02-01Z
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20220126/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20220126T194340Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Wed, 26 Jan 2022 19:43:40 GMT
Location: /test2
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 16CDE98A870299E0
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> Response Time:  1.255731ms

Bucket created successfully `minio/test2`.

The logs for the process above are still waiting on output. The process did not exit somehow.

[denizsokullu]

[harshavardhana]

The logs for the process above are still waiting on output. The process did not exit somehow.

I suspect the mc has exited already @denizsokullu there is something else going on here.

[LinAnt]

We are also experiencing this.

Like in the original issue, we are also using CircleCi. The mc command we run is:

mc cp -r -q ./path-to-thing cool-bucket/

What is interesting is that I get more output if I exec into the container:

mc: <ERROR> Failed to copy `weathericons-regular-webfont.1341a15172cec8dae06e.svg`. Object name contains unsupported characters.

[harshavardhana]

Most probably its a TTY problem - when you don't have a tty

[harshavardhana]

NOTE: We use mc in GitHub actions do not see a problem there https://github.com/minio/minio/blob/master/docs/bucket/replication/setup_3site_replication.sh

I suspect something is up with CircleCI here - @LinAnt can you confirm if your issue is seen with GitHub Actions as well?

[LinAnt]

I fired up the exact same commit workflow using https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2022-01-07T06-01-38Z and it succeeds without issue, so I still suspect there is something funky in the new mc release.

I don't run gh actions anywhere :S

[harshavardhana]

I don't see how is this an mc problem still unless we get more details here. Since they run fine in GitHub actions I don't see where there is a problem.

I tried the Docker container locally no problems were found either. Tried in k8s no problems were found either.

Beyond this, we will have to chase down what the heck CircleCI does with its TTY - this is not going to take time.