ConfigMapPropagation deletes all OwnerReferences when abandoning a copy

[grantr]

Describe the bug
When a copy ConfigMap is no longer labeled as a copy, the ConfigMapPropagation controller deletes all owner references from the ConfigMap.

Expected behavior
The ConfigMapPropagation controller deletes only the CMP owner reference from the ConfigMap.

To Reproduce
Steps to reproduce the behavior.

Knative release version
HEAD

Additional context
I expect this to be a very rare case. Hitting this seems like it would require unusual user behavior: manually adding an owner reference to a copy ConfigMap, or manually adding a copy label to an owned ConfigMap.

/cc @grac3gao

[grantr]

Any fix for this should also add a test case for abandoning a ConfigMap with multiple owner references.