[BUG] flaky test: TestRefresherCancelContext

[zhiying-lin]

did not change any refreshtoken related code, but ut failed.

=== RUN TestRefresherCancelContext
token_refresher_test.go:122:
Error Trace: token_refresher_test.go:122
Error: Test timeout
Test: TestRefresherCancelContext
Messages: TestRefresherCancelContext
--- FAIL: TestRefresherCancelContext (1.00s)

after retry, it succeeded.

Here is the detailed info.
https://github.com/Azure/fleet/actions/runs/2753311659/attempts/1
https://github.com/Azure/fleet/actions/runs/2753311659/attempts/2