-
Notifications
You must be signed in to change notification settings - Fork 19
feat: handle MC force delete #192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
d9674f2
handle MC force delete
Arvindthiru d2f3e59
garbage collect resource in fleet MC namespace
Arvindthiru ec54e7f
minor changes
Arvindthiru b4cec48
trigger reconcile on update
Arvindthiru e35a1f4
minor change
Arvindthiru 9a4f2c1
minor refactor
Arvindthiru 1450368
add suite_test file
Arvindthiru 8e849a8
add IT
Arvindthiru cd81594
revert test file changes
Arvindthiru e316128
minor refactor
Arvindthiru cbdbd1a
add comments for IT
Arvindthiru d61e2a9
minor changes
Arvindthiru d7169c2
improve IT
Arvindthiru 9f49fee
minor changes
Arvindthiru d95a2d7
minor updates
Arvindthiru f4934e6
address comments
Arvindthiru b628c5e
fix lint
Arvindthiru 8a61929
address comments
Arvindthiru bef7016
address comments
Arvindthiru e50d881
address comments
Arvindthiru 7e5bada
address comment
Arvindthiru 63351dd
lint fix
Arvindthiru b4d84a9
add UT
Arvindthiru 0ec5512
address comments
Arvindthiru b34b496
fix ut
Arvindthiru 74a493b
add more UTs for code cov
Arvindthiru e1d8525
fix UT
Arvindthiru File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| /* | ||
| Copyright (c) Microsoft Corporation. | ||
| Licensed under the MIT license. | ||
| */ | ||
|
|
||
| // Package membercluster features the MemberCluster controller for watching | ||
| // update/delete events to the MemberCluster object and removes finalizers | ||
| // on all fleet networking resources in the fleet member cluster namespace. | ||
| package membercluster | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "time" | ||
|
|
||
| "golang.org/x/sync/errgroup" | ||
| "k8s.io/apimachinery/pkg/api/errors" | ||
| "k8s.io/client-go/tools/record" | ||
| "k8s.io/klog/v2" | ||
| ctrl "sigs.k8s.io/controller-runtime" | ||
| "sigs.k8s.io/controller-runtime/pkg/client" | ||
| "sigs.k8s.io/controller-runtime/pkg/event" | ||
| "sigs.k8s.io/controller-runtime/pkg/predicate" | ||
|
|
||
| clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" | ||
| "go.goms.io/fleet/pkg/utils/controller" | ||
|
|
||
| fleetnetv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" | ||
| "go.goms.io/fleet-networking/pkg/common/hubconfig" | ||
| ) | ||
|
|
||
| const ( | ||
| ControllerName = "membercluster-controller" | ||
| ) | ||
|
|
||
| // Reconciler reconciles a MemberCluster object. | ||
| type Reconciler struct { | ||
| client.Client | ||
| Recorder record.EventRecorder | ||
| // the wait time in minutes before we need to force delete a member cluster. | ||
| ForceDeleteWaitTime time.Duration | ||
| } | ||
|
|
||
| // Reconcile watches the deletion of the member cluster and removes finalizers on fleet networking resources in the | ||
| // member cluster namespace. | ||
| func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { | ||
| mcObjRef := klog.KRef(req.Namespace, req.Name) | ||
| startTime := time.Now() | ||
| klog.V(2).InfoS("Reconciliation starts", "memberCluster", mcObjRef) | ||
| defer func() { | ||
| latency := time.Since(startTime).Milliseconds() | ||
| klog.V(2).InfoS("Reconciliation ends", "memberCluster", mcObjRef, "latency", latency) | ||
| }() | ||
| var mc clusterv1beta1.MemberCluster | ||
| if err := r.Client.Get(ctx, req.NamespacedName, &mc); err != nil { | ||
| if errors.IsNotFound(err) { | ||
| klog.V(4).InfoS("Ignoring NotFound memberCluster", "memberCluster", mcObjRef) | ||
| return ctrl.Result{}, nil | ||
| } | ||
| klog.ErrorS(err, "Failed to get memberCluster", "memberCluster", mcObjRef) | ||
| return ctrl.Result{}, err | ||
| } | ||
| if mc.DeletionTimestamp.IsZero() { | ||
| klog.ErrorS(controller.NewUnexpectedBehaviorError(fmt.Errorf("member cluster %s is not being deleted", | ||
| mc.Name)), "The member cluster should have deletionTimeStamp set to a non-zero/non-nil value") | ||
| return ctrl.Result{}, nil // no need to retry. | ||
| } | ||
|
|
||
| // Handle deleting member cluster, removes finalizers on all the resources in the cluster namespace | ||
| // after member cluster force delete wait time. | ||
| if !mc.DeletionTimestamp.IsZero() && time.Since(mc.DeletionTimestamp.Time) >= r.ForceDeleteWaitTime { | ||
Arvindthiru marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| klog.V(2).InfoS("The member cluster deletion is stuck removing the "+ | ||
| "finalizers from all the resources in member cluster namespace", "memberCluster", mcObjRef) | ||
| return r.removeFinalizer(ctx, mc) | ||
| } | ||
| // we need to only wait for force delete wait time, if the update/delete member cluster event takes | ||
| // longer to be reconciled we need to account for that time. | ||
| return ctrl.Result{RequeueAfter: r.ForceDeleteWaitTime - time.Since(mc.DeletionTimestamp.Time)}, nil | ||
| } | ||
|
|
||
| // removeFinalizer removes finalizers on the resources in the member cluster namespace. | ||
| // For EndpointSliceExport, InternalServiceImport & InternalServiceExport resources, the finalizers should be | ||
| // removed by other hub networking controllers when leaving. So this MemberCluster controller only handles | ||
| // EndpointSliceImports here. | ||
| func (r *Reconciler) removeFinalizer(ctx context.Context, mc clusterv1beta1.MemberCluster) (ctrl.Result, error) { | ||
| // Remove finalizer for EndpointSliceImport resources in the cluster namespace. | ||
| mcObjRef := klog.KRef(mc.Namespace, mc.Name) | ||
| mcNamespace := fmt.Sprintf(hubconfig.HubNamespaceNameFormat, mc.Name) | ||
| var endpointSliceImportList fleetnetv1alpha1.EndpointSliceImportList | ||
| if err := r.Client.List(ctx, &endpointSliceImportList, client.InNamespace(mcNamespace)); err != nil { | ||
| klog.ErrorS(err, "Failed to list endpointSliceImports", "memberCluster", mcObjRef) | ||
| return ctrl.Result{}, err | ||
| } | ||
| errs, ctx := errgroup.WithContext(ctx) | ||
| for i := range endpointSliceImportList.Items { | ||
| esi := &endpointSliceImportList.Items[i] | ||
| errs.Go(func() error { | ||
| esiObjRef := klog.KRef(esi.Namespace, esi.Name) | ||
| esi.SetFinalizers(nil) | ||
| if err := r.Client.Update(ctx, esi); err != nil { | ||
| klog.ErrorS(err, "Failed to remove finalizers for endpointSliceImport", | ||
| "memberCluster", mcObjRef, "endpointSliceImport", esiObjRef) | ||
| return err | ||
| } | ||
| klog.V(2).InfoS("Removed finalizers for endpointSliceImport", | ||
| "memberCluster", mcObjRef, "endpointSliceImport", esiObjRef) | ||
| return nil | ||
| }) | ||
| } | ||
| return ctrl.Result{}, errs.Wait() | ||
| } | ||
|
|
||
| // SetupWithManager sets up the controller with the Manager. | ||
| func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error { | ||
| customPredicate := predicate.Funcs{ | ||
| CreateFunc: func(e event.CreateEvent) bool { | ||
| // Ignore creation events. | ||
| return false | ||
| }, | ||
| DeleteFunc: func(e event.DeleteEvent) bool { | ||
| // trigger reconcile on delete event just in case update event is missed. | ||
| return true | ||
| }, | ||
| UpdateFunc: func(e event.UpdateEvent) bool { | ||
| // If new object is being deleted, trigger reconcile. | ||
| return !e.ObjectNew.GetDeletionTimestamp().IsZero() | ||
| }, | ||
| } | ||
| // Watch for changes to primary resource MemberCluster | ||
| return ctrl.NewControllerManagedBy(mgr). | ||
| For(&clusterv1beta1.MemberCluster{}). | ||
| WithEventFilter(customPredicate). | ||
| Complete(r) | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.