Fix #1663 Threading issue between Meter::RegisterSyncMetricStorage and Meter::Collect

[lalitb]

Fixes #1663

Changes

Add a lock guard to prevent race conditions arising due to

• Concurrent creation of sync/async instruments.
• Creation of sync/async instruments while Metric collection is ongoing.
• Concurrent collection of metrics for same meters.

These operations should ensure that sync/sync storage structures and callback registry are atomically updated.
Also added unit-test to stress meter using multiple threads.

For significant contributions please make sure you have completed the following items:

• CHANGELOG.md updated for non-trivial changes
• Unit tests have been added
• Changes in public API reviewed

[codecov]

Codecov Report

Merging #1666 (5fc0812) into main (7140166) will increase coverage by 0.79%.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1666      +/-   ##
==========================================
+ Coverage   84.97%   85.76%   +0.79%     
==========================================
  Files         170      170              
  Lines        5134     5137       +3     
==========================================
+ Hits         4362     4405      +43     
+ Misses        772      732      -40     

Impacted Files	Coverage Δ
sdk/src/metrics/meter.cc	56.16% <100.00%> (+23.09%)	⬆️
sdk/src/metrics/state/temporal_metric_storage.cc	98.22% <0.00%> (+1.79%)	⬆️
sdk/src/metrics/async_instruments.cc	100.00% <0.00%> (+20.00%)	⬆️
sdk/src/metrics/state/observable_registry.cc	80.00% <0.00%> (+22.86%)	⬆️

[lalitb]

This is ready for review now.

[esigo]

Thanks for the fix.
It would be great if you could make sure at least some of the threads finish in the unit test.
For example each thread can increment a counter when they start and decrement it when they finish.

[lalitb]

Thanks for the fix. It would be great if you could make sure at least some of the threads finish in the unit test. For example each thread can increment a counter when they start and decrement it when they finish.

Have added logic to make threads to join before finishing the unit test. Hope it suffices.