election: make sure old elec will resign

[lance6716]

What problem does this PR solve?

close #1212

What is changed and how it works?

if old election didn't resign, new election created with same session and key-prefix will always read that orphan "election key" by Observe, and can't clean it using Resign, unless TTL timeout (60s currently)

        elec := concurrency.NewElection(session, e.key)
	go func() {
		fmt.Println(elec.Campaign(ctx, e.key))
	}()
	ch := elec.Observe(ctx)
	info := <-ch
	fmt.Println("1", info)

	elec2 := concurrency.NewElection(session, e.key)
        // observe old election key
	ch2 := elec2.Observe(ctx)
	info3 := <-ch2
	fmt.Println("3", info3)
	fmt.Println(elec2.Resign(ctx))

	elec2 = concurrency.NewElection(session, e.key)
        // observe old election key
	ch2 = elec2.Observe(ctx)
	info3 = <-ch2
	fmt.Println("4", info3)
	fmt.Println(elec2.Resign(ctx))

Check List

Tests

• Unit test

Code changes

Side effects

Related changes

• Need to cherry-pick to the release branch
• Need to update release note
  fix a bug that sometimes can't evict leader of DM-master for a short time

[lance6716]

/run-all-tests

MySQL for unit test doesn't up

[GMHDBJD]

How about use debug, otherwise master will log it every TTL

[GMHDBJD]

Can we move it in front of L309 now?

[lance6716]

I guess can't. For multi-leader election, non-leader's elec.Campaign will block thus can't campaignWg.Done(). If we move campaignWg.Wait() before above two if, behaviour will change to "failed campaign won't restart a new one". Not sure this is expected

[GMHDBJD]

So the issue is because ctx done, then resign fail to remove the key?

[lance6716]

So the issue is because ctx done, then resign fail to remove the key?

I'm not sure what happened in origin issue, only found there's an orphan election key and Resign didn't cover all branch. Maybe there're other cases cause this issue 😢

[lance6716]

Should we prepare to merge this PR before 2.0 GA, or wait better locating problem to prevent bring more bug?

[lance6716]

according to logs in https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/dm_ghpr_test/detail/dm_ghpr_test/8507/pipeline/77/

I found there should be a txn deletes the old election KV

[2021-03-22T05:28:38.305Z] [2021/03/22 13:28:08.914 +08:00] [DEBUG] [interceptor.go:181] ["request stats"] [component="embed etcd"] ["start time"=2021/03/22 13:28:08.908 +08:00] ["time spent"=5.561244ms] [remote=127.0.0.1:58236] ["response type"=/etcdserverpb.KV/Txn] ["request count"=1] ["request size"=38] ["response count"=0] ["response size"=40] ["request content"="compare:<target:CREATE key:\"/dm-master/leader/12e07858679d7603\" create_revision:7 > success:<request_delete_range:<key:\"/dm-master/leader/12e07858679d7603\" > > failure:<>"]

before fail to campaign.

but for the buggy master that can't resign, there's no deleteOp

[lance6716]

can't reproduce 🤔