Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ spec:
---

## Step 4: Clean Up Resources Using the `eviction` API
- Remove the env label from the `aks-member-1` or add [a taint on the member](https://github.com/Azure/fleet/blob/main/docs/howtos/taint-toleration.md) so that the cluster won't be picked by the `clusterResourcePlacement` again.
- Remove the env label from the `aks-member-1` or add [a taint on the member](https://kubefleet-dev.github.io/website/docs/how-tos/taints-tolerations/) so that the cluster won't be picked by the `clusterResourcePlacement` again.
- Safely remove application workloads and services from `aks-member-1` once all traffic has shifted and all the client DNS caches are refreshed.

> Note: test file located [here](../testfiles/placement-eviction-member-1.yaml).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ kubectl apply -f nginx-backend-uksouth.yaml
> So that most of the traffic can be shifted to the new regions before deleting endpoints from eastus2euap region.

## Step 6: Cleanup & Finalization
- Remove the env label from the `aks-member-1` & `aks-member-3` or add [a taint on these two members](https://github.com/Azure/fleet/blob/main/docs/howtos/taint-toleration.md) so that the cluster won't be picked by the `clusterResourcePlacement` again.
- Remove the env label from the `aks-member-1` & `aks-member-3` or add [a taint on these two members](https://kubefleet-dev.github.io/website/docs/how-tos/taints-tolerations/) so that the cluster won't be picked by the `clusterResourcePlacement` again.
- To ensure the application won't be disrupted, we can create the `ClusterResourcePlacementDisruptionBudget` for the protection.

> Note: test file located [here](../testfiles/placement-disruption-budget.yaml).
Expand Down
22 changes: 21 additions & 1 deletion docs/toubleshooting/DNSBasedGlobalLoadBalancing.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,27 @@ Common reasons and solutions for `TrafficManagerBackend` not being accepted:
```
4. Not enough permissions to read the public IP address of the exported `Service` on the members.
- Ensure fleet hub networking controller has been configured correctly to access public IP address of services on the members.
5. [Reach the Azure Traffic Manager limits](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#azure-traffic-manager-limits).
5. The public IP address already exists in the Azure Traffic Manager profile.
- Please use the existing trafficManagerBackend to manage your endpoints exported by the service. It happens that you've already added
endpoints to the profile by creating the trafficManagerBackend using the same service name and profile name.
```yaml
# sample status
status:
conditions:
- lastTransitionTime: "2025-05-16T08:43:33Z"
message: "2 endpoint(s) failed to be created/updated in the Azure Traffic Manager,
for example, PUT https://management.azure.com/subscriptions/c4528d9e-c99a-48bb-b12d-fde2176a43b8/resourceGroups/zhiyinglin-fleet-dev/providers/Microsoft.Network/trafficmanagerprofiles/fleet-5abc2041-c627-4937-ab04-ffd493975adb/AzureEndpoints/fleet-390eca1c-fdb2-49c8-bf28-3e4fc2660b08#hello-world-service#dev-member-2\n--------------------------------------------------------------------------------\nRESPONSE
400: 400 Bad Request\nERROR CODE: BadRequest\n--------------------------------------------------------------------------------\n{\n
\ \"error\": {\n \"code\": \"BadRequest\",\n \"message\": \"Endpoint
target must be unique in the profile. The following endpoint target already
exists: \\/subscriptions\\/c4528d9e-c99a-48bb-b12d-fde2176a43b8\\/resourceGroups\\/mc_zhiyinglin-fleet-dev_dev-member-2_eastus2\\/providers\\/Microsoft.Network\\/publicIPAddresses\\/kubernetes-ab5eea9ca3a6d44238cf82ef2e45b41a.\"\n
\ }\n}\n--------------------------------------------------------------------------------\n; "
observedGeneration: 1
reason: Invalid
status: "False"
type: Accepted
```
6. [Reach the Azure Traffic Manager limits](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#azure-traffic-manager-limits).
- 200 endpoints are allowed per profile. If the limit is reached, consider deleting unused endpoints or requesting an increase in the limit.

Please check the `status` field of the `TrafficManagerBackend` or the `trafficmanagerbackend/controller.go` hub-net-controller-manager logs for more information.
52 changes: 51 additions & 1 deletion test/e2e/traffic_manager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,7 @@ var _ = Describe("Test exporting service via Azure traffic manager", Ordered, fu

AfterEach(func() {
By("Deleting trafficManagerBackend")
Expect(hubClient.Delete(ctx, &backend)).Should(Succeed(), "Failed to delete the trafficManagerBackend")
Expect(client.IgnoreNotFound(hubClient.Delete(ctx, &backend))).Should(Succeed(), "Failed to delete the trafficManagerBackend")
validator.IsTrafficManagerBackendDeleted(ctx, hubClient, backendName, lightAzureOperationTimeout)

By("Validating the Azure traffic manager profile")
Expand All @@ -442,6 +442,56 @@ var _ = Describe("Test exporting service via Azure traffic manager", Ordered, fu
atmValidator.ValidateProfile(ctx, atmProfileName, atmProfile)
})

It("Creating another trafficManagerBackend to export the same service", func() {
By("Creating an invalid trafficManagerBackend")
invalidBackend := wm.TrafficManagerBackend()
invalidBackend.Name = fmt.Sprintf("%s-%s", backend.Name, "invalid")
invalidBackendName := types.NamespacedName{Namespace: invalidBackend.Namespace, Name: invalidBackend.Name}
Expect(hubClient.Create(ctx, &invalidBackend)).Should(Succeed(), "Failed to create the trafficManagerBackend")

By("Validating the invalidBackend trafficManagerBackend status")
status := validator.ValidateTrafficManagerBackendIfAcceptedAndIgnoringEndpointName(ctx, hubClient, invalidBackendName, false, nil, lightAzureOperationTimeout)
validator.ValidateTrafficManagerBackendStatusAndIgnoringEndpointNameConsistently(ctx, hubClient, invalidBackendName, status)

By("Validating the Azure traffic manager profile")
// No changes should be made to the profile.
atmValidator.ValidateProfile(ctx, atmProfileName, atmProfile)

By("Deleting existing trafficManagerBackend")
Expect(hubClient.Delete(ctx, &backend)).Should(Succeed(), "Failed to delete the trafficManagerBackend")
validator.IsTrafficManagerBackendDeleted(ctx, hubClient, backendName, lightAzureOperationTimeout)

By("Validating the invalid trafficManagerBackend status and should be accepted now")
wantEndpoints := []fleetnetv1beta1.TrafficManagerEndpointStatus{
{
Weight: ptr.To(int64(50)),
Target: ptr.To(fmt.Sprintf(azureDNSFormat, memberDNSLabels[0], clusterLocation)),
From: &fleetnetv1beta1.FromCluster{
ClusterStatus: fleetnetv1beta1.ClusterStatus{Cluster: memberClusters[0].Name()},
Weight: ptr.To(int64(1)),
},
},
{
Weight: ptr.To(int64(50)),
Target: ptr.To(fmt.Sprintf(azureDNSFormat, memberDNSLabels[1], clusterLocation)),
From: &fleetnetv1beta1.FromCluster{
ClusterStatus: fleetnetv1beta1.ClusterStatus{Cluster: memberClusters[1].Name()},
Weight: ptr.To(int64(1)),
},
},
}
status = validator.ValidateTrafficManagerBackendIfAcceptedAndIgnoringEndpointName(ctx, hubClient, invalidBackendName, true, wantEndpoints, heavyAzureOperationTimeout)
validator.ValidateTrafficManagerBackendStatusAndIgnoringEndpointNameConsistently(ctx, hubClient, invalidBackendName, status)

By("Validating the Azure traffic manager profile")
atmProfile = buildDesiredATMProfile(profile, status.Endpoints)
atmProfile = *atmValidator.ValidateProfile(ctx, atmProfileName, atmProfile)

By("Deleting invalid trafficManagerBackend")
Expect(hubClient.Delete(ctx, &invalidBackend)).Should(Succeed(), "Failed to delete the trafficManagerBackend")
validator.IsTrafficManagerBackendDeleted(ctx, hubClient, invalidBackendName, lightAzureOperationTimeout)
})

It("Creating extra Azure traffic manager endpoint directly and then updating trafficManagerBackend", func() {
By("Creating a public IP address")
publicIPAddressName := fmt.Sprintf("e2e-test-public-ip-%s", uniquename.RandomLowerCaseAlphabeticString(5))
Expand Down
Loading