objstore/swift: use ncw/swift and support large files for OpenStack Swift

[FUSAKLA]

Resolves #2678
Closes #2665

As described in the issue, switch to new library to communicate with swift which supports large file types.

I also changed the maintainer of the swift integration to me since if this would be accepted, there would be not much of the old codebase left so it felt fair not to leave the burden on @sudhi-vm.

It supports also working with the large objects which was limit of the previous library.

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

• switched to new library for Swift API
• added support for large files (description in docs)
• added timeouts for Swift API

Verification

All tests are passing, tried running it as a sidecar, compactor and store for now on real workload.
Works well but will continue testing it with higher load and bigger blocks.

Eventually would like to migrate production on it.

[bwplotka]

I love it, thanks to this. Minor suggestions only.

The only concern for SWIFT is that it's not strongly consistent, are you aware of this? This makes compaction tricky?

[FUSAKLA]

thanks for the swift review 😁

I'd like to wait some time to test it out more to be sure it works fine.
Also I'd love to have some other Swift user to test it out but it seems like nobody active is here around 😕

Regarding the consistency issues.
I'm aware of Swift eventual consistency but with recently added deletion marks and delay after upload I believe it could work. My biggest concern is probably downsampling of blocks which were recently created by compaction.
But this could be mitigated by separating those by some time span? 🤔

Am I missing out some other cases where it would could blow up?

[GiedriusS]

Some minor initial comments, I haven't finished looking through the code. Thanks a lot for this. How did your testing of this go BTW?

[FUSAKLA]

Thanks for another review! :)
The testing looks promising. I see no obvious issues, but unfortunately our cluster has some issues with the load my testing recording rules are generating so there are some issues caused by this from time to time. I'm now waiting for the maintainers of the cluster to tune it up a bit so there are no errors at all and I'm 100% sure it's OK.

I'm fairly confident the uploading/reading/listing etc are working OK.
My biggest concern causing some unexpected breaking change in the configuration of the bucket.
I hope not but I failed to find any previous user of the Swift integration to verify it 😕

[FUSAKLA]

I'll rebase once #2838 to verify the tests still passes even after the refactor

[FUSAKLA]

I finally found a while to take a look at the failing tests.
Great idea with running the tests in the CI @GiedriusS , caught two issues actually (one of them regression during CR) 🎉

Now it's all green so hopefully everything is OK 🤞

I was afraid to merge this right away but it has been a while and it looks like there are no other active users of Swift so I'd go ahead and merge this if you are not against it?

[FUSAKLA]

Rebased once again ♻️

[GiedriusS]

Sorry for the delay in the review. Some questions/nits in line.

[FUSAKLA]

@GiedriusS Thanks for going through it in detail. All comments were valid and should be fixed now.

Tests related to Swift are passing but other seems to be flaky probably? 😕

PTAL

[FUSAKLA]

@GiedriusS do you think we could move on with this? I'll rebase

[hsmade]

We're bumping into the same issue. We can offer to test on our QA envs, but there we might not have big enough buckets to 'need' the splitting up on uploads.

[FUSAKLA]

@bwplotka @GiedriusS PTAL, should be up to date again
e2e tests not sure why :/

[GiedriusS]

The code looks good but I'm kind of hesitant to merge this because the new library doesn't support context.Context and that means we have to add yet another timeout parameter. @FUSAKLA perhaps you know if there are any plans to add support for contexts to the library?

[FUSAKLA]

@GiedriusS Yep, that is not great but the current state of the swift integration is almost unusable because of the file size limitation.

I opened an issue on the library and it showed up it's already considered (it was originally created before Google context library was created :)) ) so I'd send a PR there to add it.

Considering the current status of the Swift integration and how long is this PR pending, I'd be glad to merge this and ASAP send a next PR to add the context support once done in the library if you wouldn't mind. Swift is still marked as experimental in the docs so maybe the lack of context is acceptable and still having the global timeout configuration?

See the issues:
ncw/swift#161
ncw/swift#159

[bwplotka]

Happy with this approach given the inusability of previous integration. Thanks for this work and sorry for lag in review.

Good work!