[BUG] resource not be deleted after remove the cluster from CRP's policy

[mingqishao]

Describe the bug

After remove the member cluster from CRP's policy, the propagated resources not be deleted from member cluster

Environment

Please provide the following:

• Fleet Resource ID: /subscriptions/3959ec86-5353-4b0c-b5d7-3877122861a0/resourceGroups/minsha-test-work-2/providers/Microsoft.ContainerService/fleets/minsha-test-work-2
• Hub cluster resource ID: /subscriptions/3959ec86-5353-4b0c-b5d7-3877122861a0/resourcegroups/FL_minsha-test-work-2_minsha-test-work-2_northeurope/providers/Microsoft.ContainerService/managedClusters/hub
• member cluster resource ID: /subscriptions/3959ec86-5353-4b0c-b5d7-3877122861a0/resourcegroups/minsha-test-work-2/providers/Microsoft.ContainerService/managedClusters/aks-member-4

To Reproduce

Steps to reproduce the behavior:

1. create a Fleet and 3 members (aks-member-1, aks-member-2, aks-member-3).
   2.create a CRP and policy included a not exists member cluster, aks-member-4

apiVersion: fleet.azure.com/v1alpha1
kind: ClusterResourcePlacement
metadata:
  name: crp1
spec:
  resourceSelectors:
    - group: ""
      version: v1
      kind: Namespace
      name: app 
    - group: rbac.authorization.k8s.io
      version: v1
      kind: ClusterRole
      labelSelector:
        matchLabels:
          fleet.azure.com/name: test
  policy:
    clusterNames:
      - aks-member-1
      - aks-member-2
      - aks-member-4

3. create ClusterRole resource in hub. This resources was propagated to aks-member-1 and aks-member-2

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: test-role
  labels: 
    fleet.azure.com/name: test
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [ "" ]
    resources: [ "events" ]
    verbs: [ "get", "list", "watch", "create", "patch" ]
  - apiGroups: [ "" ]
    resources: ["nodes"]
    verbs: [ "get", "list", "watch"]

4. create a new AKS cluster (aks-member-4) and join fleet. Once the member cluster joined hub cluster, the ClusterRole resource, test-role, was propagated to aks-member-4.
5. Update the CRP, remove aks-member-4 from the policy

apiVersion: fleet.azure.com/v1alpha1
kind: ClusterResourcePlacement
metadata:
  name: crp1
spec:
  resourceSelectors:
    - group: ""
      version: v1
      kind: Namespace
      name: app 
    - group: rbac.authorization.k8s.io
      version: v1
      kind: ClusterRole
      labelSelector:
        matchLabels:
          fleet.azure.com/name: test
  policy:
    clusterNames:
      - aks-member-1
      - aks-member-2

Expected behavior

After the ask-member-4 be removed from policy, the ClusterRole resource, test-role, expect be removed from aks-member-4 member cluster. But that still there.

Screenshots

Here is from hub cluster:

Mitchs-MacBook-Pro:qa minsha$ kubectl get works -A
NAMESPACE                   NAME   AGE
fleet-member-aks-member-1   crp1   4h23m
fleet-member-aks-member-2   crp1   3h59m

Mitchs-MacBook-Pro:qa minsha$ kubectl get memberclusters
NAME           JOINED   AGE
aks-member-1            6h26m
aks-member-2            6h18m
aks-member-3            6h6m
aks-member-4            4h42m

Mitchs-MacBook-Pro:qa minsha$ kubectl get crp crp1 -o yaml
apiVersion: fleet.azure.com/v1alpha1
kind: ClusterResourcePlacement
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"fleet.azure.com/v1alpha1","kind":"ClusterResourcePlacement","metadata":{"annotations":{},"name":"crp1"},"spec":{"policy":{"clusterNames":["aks-member-1","aks-member-2"]},"resourceSelectors":[{"group":"","kind":"Namespace","name":"app","version":"v1"},{"group":"rbac.authorization.k8s.io","kind":"ClusterRole","labelSelector":{"matchLabels":{"fleet.azure.com/name":"test"}},"version":"v1"}]}}
  creationTimestamp: "2022-09-02T23:57:31Z"
  generation: 7
  name: crp1
  resourceVersion: "248611"
  uid: 0d059dbf-5e4f-41f2-b44e-ccc3a9bb5075
spec:
  policy:
    clusterNames:
    - aks-member-1
    - aks-member-2
  resourceSelectors:
  - group: ""
    kind: Namespace
    name: app
    version: v1
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    labelSelector:
      matchLabels:
        fleet.azure.com/name: test
    version: v1
status:
  conditions:
  - lastTransitionTime: "2022-09-03T00:01:48Z"
    message: Successfully scheduled resources for placement
    observedGeneration: 7
    reason: ScheduleSucceeded
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-09-03T00:25:45Z"
    message: Successfully applied resources to member clusters
    observedGeneration: 7
    reason: ApplySucceeded
    status: "True"
    type: Applied
  selectedResources:
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: test-role
    version: v1
  targetClusters:
  - aks-member-1
  - aks-member-2

Here is from aks-member-4 member cluster

Mitchs-MacBook-Pro:qa minsha$ kubectl get clusterroles test-role -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    fleet.azure.com/last-applied-configuration: '{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"fleet.azure.com/spec-hash":"b23aafbc34cb0ffbda43d739f6c49513a524a6d87b39a56d5e205bf8033d0b21"},"labels":{"fleet.azure.com/name":"test"},"name":"test-role","ownerReferences":[{"apiVersion":"multicluster.x-k8s.io/v1alpha1","blockOwnerDeletion":false,"kind":"AppliedWork","name":"crp1","uid":"a853c4ac-b0c1-48ac-b2e1-cc6fb697525c"}]},"rules":[{"apiGroups":[""],"resources":["secrets"],"verbs":["get","list","watch"]},{"apiGroups":[""],"resources":["events"],"verbs":["get","list","watch","create","patch"]},{"apiGroups":[""],"resources":["nodes"],"verbs":["get","list","watch"]}]}'
    fleet.azure.com/spec-hash: b23aafbc34cb0ffbda43d739f6c49513a524a6d87b39a56d5e205bf8033d0b21
  creationTimestamp: "2022-09-03T00:01:49Z"
  labels:
    fleet.azure.com/name: test
  name: test-role
  ownerReferences:
  - apiVersion: multicluster.x-k8s.io/v1alpha1
    blockOwnerDeletion: false
    kind: AppliedWork
    name: crp1
    uid: a853c4ac-b0c1-48ac-b2e1-cc6fb697525c
  resourceVersion: "28561"
  uid: c030c370-1d67-4341-bdd5-1a825ed4899f
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - get
  - list
  - watch
  - create
  - patch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch

Mitchs-MacBook-Pro:qa minsha$ kubectl get appliedworks crp1 -o yaml
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: AppliedWork
metadata:
  creationTimestamp: "2022-09-03T00:01:48Z"
  generation: 1
  name: crp1
  resourceVersion: "28562"
  uid: a853c4ac-b0c1-48ac-b2e1-cc6fb697525c
spec:
  workName: crp1
  workNamespace: fleet-member-aks-member-1
status:
  appliedResources:
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
    name: test-role
    ordinal: 0
    resource: clusterroles
    uid: c030c370-1d67-4341-bdd5-1a825ed4899f
    version: v1

Additional context

Add any other context about the problem here.

[mingqishao]

I think that is my mistake. I download the wrong kubeconfig for the aks-member-4 so I watch the wrong result. I will verify it again.

[mingqishao]

This is a mistake of copying kubeconfig. close this issue.