[Bug]: Updating from 1.19.1 to 1.20.1 breaks lambda with older versions

[woehrl01]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

No response

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

kubectl get functions.lambda.aws.upbound.io 

Error from server: conversion webhook for lambda.aws.upbound.io/v1beta1, Kind=Function failed: cannot convert from the spoke version "v1beta1" to the hub version "v1beta2": cannot convert the map[string]any representation of the conversion target back to the object itself: cannot restore struct from: slice

What happened?

Can't list resources anymore, which breaks kubectl and others tools e.g. ArgoCD

Relevant Error Output Snippet

Crossplane Version

1.17.1

Provider Version

1.20.1

Kubernetes Version

No response

Kubernetes Distribution

No response

Additional Info

No response

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!

[nsaini-figma]

I am running into this again but intermittently. Somehow the conversion webhook pod gets into a bad state and has to be restarted, which fixes the issue. Any ideas what could be causing this?

[RanMarkovich]

This also happened to me, crossplane-version-1.18.2 with Provider version 1.20.0
Removing the provider CRD and reinstalling it resolves the issue.

[nsaini-figma]

@RanMarkovich You mean the CRD itself from the cluster or just re-installing the provider deployment?

[ulucinar]

Hi @RanMarkovich,

This also happened to me, crossplane-version-1.18.2 with Provider version 1.20.0

Did you also encounter this issue as you upgraded from crossplane-contrib/provider-upjet-aws@v1.19.1 to crossplane-contrib/provider-upjet-aws@v1.20.0?

[jonathan-innis]

Hi @ulucinar. Yes, we are running xpkg.upbound.io/upbound/provider-aws-lambda:v1.20.0 and xpkg.upbound.io/upbound/provider-family-aws:v1.20.0 and running into this issue.

[jonathan-innis]

I'm suspicious that there is some weird interaction with SSA since I'm seeing issues like crossplane/upjet#465 crop-up that seem to be related to similar conversion failures.

[jonathan-innis]

Also, an interesting behavior to note here is that restarting the controller seems to fix the intermittent issue that we run into. It's not clear why that's the case -- it seems like there is potentially some state being held by the conversion webhook that is causing this issue.