[Bug]: Azure CopyObject uses async StartCopyFromURL without polling copy status — snapshot restore can report success before data is copied

[huanghaoyuanhhy]

Environment

• Milvus Version: master @ 24e43c6 (snapshot restore via server-side copy)
• Deployment Mode: cluster / standalone
• MQ: n/a
• SDK: n/a (reproducible via Go/Python SDK + Azure REST)
• OS: n/a

Reproduction

Option B: Steps (systemic / timing issue)

1. Deploy: Milvus with object storage provider azure (any account config; same-account copy)
2. Setup: create a collection with a large vector field, insert enough data to produce
   multi-GB sealed segments, flush, build index
3. Workload: call CreateSnapshot then RestoreSnapshot, wait for the restore job to
   report success (GetRestoreSnapshotState → completed)
4. Trigger: immediately after restore reports success, call Get Blob Properties
   (HEAD) on the destination blobs under the restored collection's path
5. Observed via: destination blob response headers — x-ms-copy-status,
   x-ms-copy-progress, x-ms-copy-id; then load/query the restored collection

Trigger Conditions

• Frequency: always, on Azure, for blobs large enough that the background copy
  doesn't finish within the restore job's copy loop
• First observed after: restore of large segments (copy takes seconds to minutes)
• Does NOT happen when: provider is S3/MinIO or GCS (synchronous copy APIs);
  on Azure only when the background copy happens to finish before any consumer
  touches the destination

Expected Behavior

ChunkManager.Copy / ObjectStorage.CopyObject on Azure should behave like the
S3/GCS implementations: return only after the object is fully copied (or fail with
an error). Restore must not report a segment as copied while its files are still
being copied in the background.

Actual Behavior

AzureObjectStorage.CopyObject (internal/storage/azure_object_storage.go:198)
calls StartCopyFromURL — Azure's asynchronous Copy Blob API, which returns
202 Accepted immediately with x-ms-copy-status: pending — and returns nil without
polling the copy status. Consequences:

1. Restore (copy-segment job) reports success while segment files are still copying
   in the background; the destination is a zero-length/partial blob until the copy
   finishes. A read of restored data can return partial or empty content.
2. If the source blob is modified during the copy, Azure fails the copy in the
   background (x-ms-copy-status: failed); Milvus never notices and there is no
   retry. A pending copy also times out after two weeks, leaving an empty blob.
3. The destination cannot be written/snapshotted while a copy is pending (409
   Conflict).

No error is raised in logs — this is a silent correctness issue.

Error Logs

No error in logs, results are incorrect. The failure is only visible on the Azure
side via Get Blob Properties:

x-ms-copy-id: <id>
x-ms-copy-status: pending   (or failed)
x-ms-copy-progress: <bytes copied>/<total>
x-ms-copy-status-description: <error, when failed>

Non-default Configuration

minio:
  provider: azure

Analysis Hints

• Suspect code: internal/storage/azure_object_storage.go:198 (CopyObject →
  StartCopyFromURL, no status polling)
• Call chain: internal/datanode/importv2/copy_segment_utils.go:144 (copyFile) →
  internal/storage/remote_chunk_manager.go:445 (Copy) → CopyObject. No
  read/stat/checksum verification of destination files after the copy loop
  (CopySegmentAndIndexFiles only builds metadata afterwards)
• Contrast: S3 (minio_object_storage.go:115) and GCS
  (gcp_native_object_storage.go:161) use synchronous copy APIs and are not
  affected
• Azure docs: Copy Blob
  — "can finish asynchronously... Poll the destination blob by using Get Blob
  Properties to examine the x-ms-copy-status header until the operation finishes
  or fails"; block blob contents are visible via Get only after the copy finishes;
  Get Blob Properties
  — x-ms-copy-status: <pending | success | aborted | failed>; pending copies
  time out after two weeks
• Suggested fix: either (a) use the synchronous CopyFromURL API (256 MiB limit)
  with a StageBlockFromURL multipart fallback for larger objects, or (b) poll
  GetProperties for x-ms-copy-status: success after StartCopyFromURL,
  honoring context cancellation and retrying on failed