Upsert Resource on Adopt-or-Create

[JonathanGraniero]

Describe the bug
When using specifically the adopt-or-create annotation, there is some unexpected behavior when trying to adopt an existing queue. The expected functionality is that if we are using adopt-or-create and not marking the resource as read-only, the CR should become the source of truth and upsert when adopting the resource.

I have an SQS queue that is currently deployed that has some different configurations than what what my CR has. There are two fields, one is visibilityTimeout and one is the policy field. I essentially have noticed that our deployed resources have some issues that I'm looking to standardize so i'm expecting these differences.

When I deploy the CR in the cluster, I'm getting errors trying to adopt the resource:

operation error SQS: CreateQueue, https response error StatusCode: 400, RequestID: e1b5bf5b-142e-5f3a-84b4-dfc59989e68c, QueueAlreadyExists: A queue already exists with the same name and a different value for attribute VisibilityTimeout

Steps to reproduce

Create a queue that has a visibility timeout and/or resource policy that you're looking to change, have this be unmanaged (I had it created via terraform). In my example, the deployed visibility timeout uses 300.

Create a CR using adopt-or-create annotation and does not include adoption-fields annotation:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: '-1'
  services.k8s.aws/adoption-policy: adopt-or-create
  finalizers:
    - finalizers.sqs.services.k8s.aws/Queue
  generation: 2
  labels:
    app: my-app
    app.kubernetes.io/instance: my-app
    queue-type: primary
  name: my-queue
  namespace: namespace
spec:
  delaySeconds: '0'
  maximumMessageSize: '262144'
  messageRetentionPeriod: '1209600'
  queueName: my-queue
  receiveMessageWaitTimeSeconds: '0'
  visibilityTimeout: 30

When this deploys in the cluster, I'm getting a reconcile error:

operation error SQS: CreateQueue, https response error StatusCode: 400, RequestID: e1b5bf5b-142e-5f3a-84b4-dfc59989e68c, QueueAlreadyExists: A queue already exists with the same name and a different value for attribute VisibilityTimeout

I have attempted to switch this to using adopt and then supplying adoption-fields to see if I can work around this, but when I use the adopt tag instead of adopt-or-create and provide the adoption-fields property, I'm still seeing the same error as above.

Expected outcome
I'm expecting this to be an upsert, so that if the adopted resource has values different than what is deployed, it'll update the resource to match the manifest, rather than throwing an error saying that there is a mismatch.

Environment
ACK Versions:
Helm 46.25.7
SQS Controller public.ecr.aws/aws-controllers-k8s/sqs-controller:1.1.11 (which I think should have the recent release for this fix https://github.com/aws-controllers-k8s/sqs-controller/releases/tag/v1.1.11 -> https://github.com/aws-controllers-k8s/runtime/releases/tag/v0.46.1)

• Kubernetes version: 1.30
• Using EKS (yes/no), if so version? yes, eks.30
• AWS service targeted (S3, RDS, etc.) SQS, but also seeing it in S3 depending on the props, I'm more concerned about SQS than s3 through at this current moment

[github-actions]

Hello @JonathanGraniero 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

Hi @JonathanGraniero
I just attempted Upsert with the latest SQS release, and was able to successfully update the AWS resource right after create.
https://github.com/aws-controllers-k8s/sqs-controller/releases/tag/v1.1.11
Here's the manifest I used

apiVersion: sqs.services.k8s.aws/v1alpha1
kind: Queue
metadata:
  name: basic-queue-das
  annotations:
    services.k8s.aws/adoption-policy: "adopt-or-create"
    services.k8s.aws/deletion-policy: "retain"
    services.k8s.aws/adoption-fields: | 
        {
          "queueURL": "https://sqs.us-west-2.amazonaws.com/399481058530/myqueue"
        }
spec:
  queueName: myqueue
  delaySeconds: "0"
  visibilityTimeout: "30"

Please let me know if you still see the same issue.

[JonathanGraniero]

You're using adoption fields though, can you try without it?

[JonathanGraniero]

This is also with adopting an existing resource, not creating a new one.

[michaelhtm]

Yes, I am attempting to adopt an existing resource.
To get a specific queue, you need to provide the QueueURL.
This is the API we use to adopt an existing Queue: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html

https://github.com/aws-controllers-k8s/sqs-controller/blob/20ba16b6eb8becc8a6cd10f227a78824f766897b/pkg/resource/queue/sdk.go#L210-L226

Without a queueURL, the controller won't be able to find which queue you're trying to adopt.