Fix potential deadlock when k8s client is used

[skonto]

Changes

• Removes the global metrics config mutex to avoid a potential deadlock when k8s client is used due to the
  registered metrics hooks in the client.
• This PR preserves the previous semantics when managing the metrics config etc so it should be transparent from a user point of view.
• It follows the worker approach in the opencensus lib.

/kind bug

Fixes #807

/cc @vagababov @evankanderson

[codecov]

Codecov Report

Merging #2031 (3ca75a4) into master (29092fe) will increase coverage by 0.03%.
The diff coverage is 85.96%.

@@            Coverage Diff             @@
##           master    #2031      +/-   ##
==========================================
+ Coverage   67.15%   67.19%   +0.03%     
==========================================
  Files         213      214       +1     
  Lines        8991     9019      +28     
==========================================
+ Hits         6038     6060      +22     
- Misses       2682     2686       +4     
- Partials      271      273       +2     

Impacted Files	Coverage Δ
metrics/metrics_worker.go	71.42% <71.42%> (ø)
metrics/exporter.go	84.61% <100.00%> (+6.35%)	⬆️
metrics/testing.go	100.00% <100.00%> (ø)
test/gcs/mock/mock.go	91.39% <0.00%> (-2.16%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 29092fe...908eed3. Read the comment docs.

[skonto]

TestMetricsExport fails again.

[vagababov]

I think this is already called due to init?

[skonto]

Nope that was really weird because I thought init from the metrics package (since it is imported) it would be call first but it does not. It fails with a nil pointer exception when it tries to access the mWorker instance.
Let me double check maybe something messed up.

[skonto]

Cant reproduce it will update.

[skonto]

@evankanderson gentle ping.

[evankanderson]

Other than the unit test failure, this looks reasonable to me.

[evankanderson]

I think this can be return <-updateCmd.done

[evankanderson]

Running main and your branch 84 times, I see about an equivalent number of failures (8 vs 9(, but there seems to be a lot more output on failure with this PR, and I see a timeout/missing data failure in the OpenCensus export in addition to the e2e_test.go:320 timeout on Prometheus errors. I'm willing to believe that the OpenCensus failure was a fluke and that this doesn't make the Prometheus failures any worse (though they seem much higher than I'd expect).

/retest
/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: evankanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• metrics/OWNERS [evankanderson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[skonto]

/retest

[skonto]

@evankanderson I have seen that test failing multiple times in my other pr #2005, left a comment there, so I am not sure if this is any different. I will try to debug that and see if it can be fixed in another PR (I am out this week btw).