Track GPU memory and runtime and arbitrary metrics in tests

[terrykong]

Track the following over time for unit/functional tests

• runtime
• peak memory usage
• test-specific metrics (for thresholded tests) like
  • logprob error
  • loss