aws: fix assertion failure in debug due to timer access outside of main thread

[nbaws]

Commit Message: aws: fix assertion failure in debug due to timer access outside of main thread

Additional Description:

Patch to ensure async credential providers do not access main thread timers when credentials are requested. This was a larger fix than originally expected, due to substantial test case rewrites, changes to refresh logic and changes to the cluster initialisation sequence.

This patch performs the following:

• Changes async providers to kick off credential refresh process via onClusterAddOrUpdate, replacing the init handler
• Changes async providers to be truly async. They now perform credential refresh based on a timer, set to the expiration time returned in the credential payload, or to the default cache duration of 1 hour. Async providers will no longer trigger credential refresh via the data plane (which was actually the cause of the original bug). Async providers will start refreshing with a 2 second timer and doubling to 32 seconds (or until they succeed) avoiding load on STS or IMDS.
• Fixes the cache duration calculation to be actually 1 hour, rather than 1 hour * 60 * 60 :)
• Fixes a reported bug that also caused assertion failure in route specific configuration
• Async providers now honour any expiration provided from their credential source
• Async providers now have statistics to capture number of success/failed refreshes, used in test cases
• Webidentity credential provider now handles the (unlikely) case where more than one region is present in the configuration, such as multiple route specific configs. Webidentity sts clusters will have the region name appended.

Risk Level: Low
Testing: Unit
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue] #33962
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

[nbaws]

@suniltheta

[nbaws]

A note on this PR - the majority of the change is to implement a handler for onClusterAddOrDelete, and that handler literally calls refresh() once the cluster is ready to handle requests and is never used again. If there is a shorter and/or cleaner way to do this, please let me know. I've used code from dynamic forward proxy to do most of the onClusterAddOrDelete handling.

[mattklein123]

@suniltheta for first pass, thanks.

/wait

[suniltheta]

thanks for making this change. I am still in process of reviewing tests. Though I would leave f/b as I have them.

[nbaws]

/retest

[nbaws]

@suniltheta some minor refactoring - there was an initialization issue I found with upstream filter that I've now addressed.