Grace period and leader election configuration

[ryanzhang-oss]

Describe the bug

We need to configure the leader election lease duration/renew time along with the graceful terminating period time (https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) so that we get an optimal configuration for the production.

Environment

Please provide the following:

• Hub cluster details
• Member cluster details

To Reproduce

Expected behavior

1. no two controllers handle the events at the same time.
2. the leader duration is set long enough that a pod is terminated before.
3. the leader duration is set to short enough that there is no long waiting between leader exchange.
4. the leader duration is set to long enough that we don't lose the lease because of network or other glitches.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.