apiv2: "podman info" is slowing down

[edsantiago]

I've seen several CI flakes in test-apiv2 that look like:

not ok 46 [01-basic] Time for ten /info requests
#  expected: <= 5 seconds
#    actual: 6 seconds

e.g. in pr 5388. Rerunning the test usually clears it up.

I could easily bump up the time limit, but I put that test in there for a reason: the API was getting slow. I think 6 seconds for 10 requests is a little concerning. @jwhonce @baude could you look into this please?

[cevich]

When running for CI purposes, it's really difficult to maintain control over precisely what the base-line expected performance should be. Therefor, it's potentially challenging to maintain performance-tests among what's normally a binary pass/fail outcome for development purposes.

Here are two ideas I've come across over my years (good/bad take/leave as you will):

1. Determine a "good enough" (imperfect) method of determining the base-line CPU performance (all else being equal), then scale the expected-timeout of the test accordingly. For example, based on a value derived from bogomips (/proc/cpuinfo) divided by some invented baseline number, then multiplied by the number of CPUs. Perhaps further scale the bogo-cpus factor, by the inverse of: the 8th column of /proc/stat (steal time) divided by some number related to $(getconf CLK_TCK) (?)

2. Add a separate Cirrus-CI task dedicated to only running performance-related tests, where the task can more precisely specify CPU/memory resources (separate from the otherwise binary-result tasks). Ref: Specify Machine Type. Perhaps execute this task on a timer rather than for every PR, and have it notify failures via e-mail or IRC or slack or whatever.

Doing both is also an option. In any case, the goal is to have automated tests not block rapid development as much as is possible, but also notice gross performance regressions. Actionable performance-testing results nearly always come down to controlling the baseline and/or using measurements in very scientific/precise way. A challenging task, but without which, the measurement reliability can always be questioned, including (right/wrong) calls to "just disable the test" in times of panic 😞

[abitrolly]

@edsantiago it may happen that image is scanned several times even for a single request.

[jwhonce]

I walked the code and nothing obvious changed. What format are you requesting?

[edsantiago]

This is a trivial GET /v1.40/info, repeated ten times

[jwhonce]

It is something in the environment, on my laptop I can serve 50 calls in < 6 seconds. I did see where we need to read all the containers for state, that could create an O(n) time against the number of containers. How many containers existed at the time? podman ps --all

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago @jwhonce Does this issue still exists?

[edsantiago]

I don't know - I haven't been diligent about tracking flakes. Closing - if/when it becomes a problem again I'll just open a PR to bump up the timeout.