fix: split migrate-to-v1.sh into pre/post phases to fix username backfill

[barunavo]

Summary

Fixes #163.

Update 3: the previous update's fixes only made silent overwrites fail
loudly after attempting a patch. That's still the wrong default for a
genuinely live claim - one actively reconciled by GitOps or a still-busy
owning composite - where the right move is to not touch it at all rather
than gamble on the race not landing.

pre already records exactly this case: if a CR or its owning composite
doesn't confirm ReconcilePaused within the timeout, it's logged to
unpaused-owners.txt. post now actually reads that file (is_unpaused())
before processing each User/AdvancedCluster, and skips it entirely -
still recording the skip in the same failure list that blocks proceeding to
install v1.x - instead of patching and relying on the post-verification
alone. Unit-tested directly against the three cases (CR listed by name, CR
whose owner is listed, CR that's neither).

Update 2: live testing against a real cluster (not the maintainer's, a
different one) surfaced two more real gaps, fixed in the latest commit:

1. Post-phase Step 1 never verified the username patch actually stuck.
   In one observed run, kubectl patch reported success and
   authDatabaseName persisted correctly from the same patch call, but
   username came back empty on a fresh get - a config-error we haven't
   fully root-caused (the owning composite was independently confirmed to
   be genuinely paused at the time, ruling out the composite-overwrite race
   the previous update in this PR described - so it isn't that, at least
   not in every case). Rather than guess further, Step 1 now re-reads the
   object after patching and fails loudly - listing every affected CR - if
   it doesn't match, instead of reporting success either way. This turns
   any future occurrence of this (whatever the exact cause) into a clear,
   actionable failure instead of a silent one that only surfaces once
   v1.x's CEL validation rejects the CR.

   If you're already stuck with a CR in this state: the CRD conversion
   itself did finish correctly in the case we looked at (it now serves
   only v1alpha3), so a direct kubectl patch ... -p '{"spec":{"forProvider":{"username":"<name>"}}}' should work today -
   the schema has the field now, and if the owning composite is genuinely
   paused there's nothing left to race with. Verify with a fresh get
   afterward regardless.

2. Post-phase Step 2 only touched AdvancedCluster CRs, assuming CRD
   conversion would reshape the object. It doesn't: this provider has no
   conversion webhook, so Kubernetes' default (None) conversion is a pure
   passthrough. electableSpecs, autoScaling, analyticsAutoScaling,
   readOnlySpecs and analyticsSpecs are all arrays-of-one-object in
   v1alpha2 and plain objects in v1alpha3 - a bare touch gets the
   mismatched array pruned to nothing by structural-schema validation,
   losing every field in it (not just failing to add the new diskSizeGb
   location - instanceSize/nodeCount/etc. are lost too). The same touch
   also can't backfill v1alpha3's new required spec.forProvider.name,
   which has no v1alpha2 equivalent at all and is enforced by a CEL rule.

   Step 2 now reads each CR's pre-migration shape from the pre-phase's own
   backup, builds the reshaped v1alpha3 structure explicitly (moving
   diskSizeGb into electableSpecs/readOnlySpecs/analyticsSpecs,
   converting the five array fields to objects), derives name from the
   CR's own external-name (matching the "<name>:<cluster_id>" format
   config/mongodbatlas/config.go already uses for this resource), and
   verifies the patch persisted the same way Step 1 does.

Verified end-to-end against a real apiserver (kind, not mocked) using the
actual v0.4.1 and v1.0.0 CRDs: pre + post now leave both a
DatabaseUser and an AdvancedCluster fully valid under v1alpha3, with
diskSizeGb correctly relocated and no data loss in
electableSpecs/autoScaling.

---

Update 1 (previous): the first version of this PR (commit 0cb64f3) turned out to be insufficient. Testing it end-to-end against a real cluster (not just dry-run) surfaced two further problems, fixed in that revision:

1. The same chicken-and-egg problem, just relocated. spec.forProvider.username doesn't exist in the v0.x (v1alpha2) schema — only in v1.x's v1alpha3, which replaces v1alpha2 rather than adding alongside it. Moving the backfill to a post phase (after installing v1.x) doesn't help: Kubernetes flatly refuses to install a CRD that drops a version still listed in status.storedVersions. There's no point in the naive sequence where the field can be written. The only way through is a transitional CRD — built by merging the live CRD with the target v1.x CRD's version block, applied directly via kubectl apply (bypassing Crossplane's package manager, which only ships the final single-version CRD) — that serves both v1alpha2 and v1alpha3 side by side. This lets existing objects be re-stored at v1alpha3 (and username backfilled) before v1alpha2 is dropped and the real v1.x provider is installed.

2. Pausing the leaf CR isn't enough. If a Crossplane Composition owns the User/AdvancedCluster CR (the normal way these are consumed on a real platform), pausing the CR only stops the provider's reconciler — it does not stop the owning Composite Resource (XR) from continuing to render and re-apply its (v1alpha2-shaped, username-less) base template over it, silently wiping the backfilled field on the next reconcile. Confirmed live: backfilled username, moved on, came back later, and it was gone — the owning XR's ComposeResources had overwritten it in between. The script now discovers each CR's controlling owner (ownerReferences[] where controller: true) and pauses that too.

What changed

scripts/migrate-to-v1.sh now takes an explicit phase argument and a --v1-crds-dir:

./scripts/migrate-to-v1.sh pre --v1-crds-dir /path/to/v1.x/package/crds
./scripts/migrate-to-v1.sh post
# install/activate the v1.x provider package here
# unpause CRs and their owning composites

• pre: backs up CRs, pauses each CR and its owning composite (if any), builds and applies a transitional dual-version CRD from the live CRD + the v1.x CRD YAML you provide.
• post: backfills username/authDatabaseName, touches AdvancedClusters to trigger re-storage at v1alpha3, removes v1alpha2 from status.storedVersions, then drops v1alpha2 from spec.versions entirely so the CRD matches what the real v1.x package will install.

Also fixed in passing: the original script computed existing_auth_db but never included it in the patch, despite the docs claiming it does.

docs/migration-to-v1.md rewritten to explain the transitional-CRD rationale and the composite-pausing requirement.

Test plan

Verified live, end-to-end, against a real cluster running v0.4.1 with pre-existing User/AdvancedCluster CRs owned by Crossplane Compositions (not synthetic test data):

• Built and applied a transitional CRD (v1alpha2 + v1alpha3, v1alpha3 as storage) for both users.database.mongodbatlas.crossplane.io and advancedclusters.mongodbatlas.crossplane.io — confirmed Crossplane's provider-revision controller does not fight or revert a hand-applied CRD change.
• Backfilled username against the live v1alpha3 schema — confirmed it's silently dropped by the unpaused owning XR's next reconcile (reproducing problem Set default external name config as IdentifierFromProvider and a common referencer for project_id #2 above), and confirmed it sticks once the owning XR is actually paused.
• Removed v1alpha2 from storedVersions, then dropped it from spec.versions — confirmed the real v1.x provider package then activates Healthy with no CRD-establishment error (previously failed with cannot establish control of object: ... v1alpha2 was previously a storage version ...).
• Confirmed the migrated User CR reconciles cleanly (SYNCED=True, READY=True) under the real, running v1.0.0 provider controller.
• Rolled the same cluster all the way back to a genuine pre-migration v0.4.1 state afterward (reverse transitional CRD, re-storage at v1alpha2, spec.versions/storedVersions cleanup) and confirmed the User CR resyncs cleanly under v0.4.1 too — proving the transitional-CRD technique is safe in both directions.
• bash -n scripts/migrate-to-v1.sh (syntax check)
• DRY_RUN=true ./scripts/migrate-to-v1.sh pre --v1-crds-dir ... correctly discovers and would-pause each CR's actual owning composite kind (verified against 4 different real XR kinds on the test cluster)

Not verified: --v1-crds-dir pointed at CRDs for resources with namespaced (*.m.crossplane.io) variants beyond users/advancedclusters — the script's AFFECTED_CRDS list still only names these two resource families explicitly.