GC Containers and volumes when present in the worker but not in the DB

[cirocosta]

Hey,

Once we got #2588 in, we added support for GCing when there's a mismatch between what is in the worker and what is in the DB when

• DB has containers/volumes that a worker doesn't have,

But we don't do the other "side" of the diff:

• a worker having containers/volumes that the DB doesn't have.

This means that if worker-0, for some reason, has volumes that the DB doesn't have, those volumes will never be cleaned up unless someone manually discovers that diff and performs the removals.

A scenario where this could occur:

• the user takes backup of the DB at t0,
• time goes on ... the worker has some volumes at t1,
• DB gets restored to t0,
• the worker will forever have the extra volumes there.

Do you think this could have repercussions in the way we create containers/volumes? E.g., in the case of ephemeral check containers?

Thanks!

[vito]

Yeah, we'd have to be careful not to remove ephemeral or health-check containers/volumes. Maybe this could be done by marking the containers as ephemeral/health-check via properties?

[cirocosta]

Hey,

Another that @YoussB and I were discussing:

• worker launched with --ephemeral starts
• containers and volumes are created
• network problems happen, worker "stalls", thus, gets removed from the db
• worker comes back, but its old volumes are never deleted

Does that make sense?

Thanks!

[cirocosta]

(related: #2835)

[YoussB]

@ddadlani and I saw the same issue when working with a customer that caused a big problem since most of the workers got overloaded by containers after an upgrade.

[pivotal-jamil-shamy]

@ddadlani and I saw the same issue when working with a customer that caused a big problem since most of the workers got overloaded by containers after an upgrade.

@ddadlani @YoussB @cirocosta

To add more context for reference:

1. We were running concourse 5.2.1.
2. The average container count per worker before the upgrade was 100
3. We upgraded to concourse 5.5.1
4. We noticed that the container count per workers started to climb sharply.
5. After a few hours of finishing the upgrade, all the workers' container count was maxed out at 256.
6. No more builds were able to be scheduled.
7. After investigation, we found that there are many containers on the workers were not "known" by the DB, i.e. they were orphaned.
8. The only possible solution we were left with is to recreate ALL the workers VMs.

[pivotal-jamil-shamy]

To reiterate what @ddadlani said in #4500 , I think it would be great to know why these containers are not being cleared upon an upgrade

[kcmannem]

There is an edge case we are currently trying to reproduce:
In Concourse whenever a heartbeat from a worker does not contain handles the ATC tracks in the db. We start a countdown (default 5 mins), where if subsequent heartbeats still do not contain the handle in question. We delete the that container row in the db regardless of if it is in the "created" state. If somehow this handle starts to appear in the heartbeat to the ATC again, the ATC will ignore it as it only maintains the lifecycle of containers in its db, it assumes that someone else created the container so it doesn't need to claim responsibility of the reappearing containers lifecycle.

This can happen if the container goes missing in a heartbeat first and then some network interruption stalls the worker for longer than 5 mins. The GC will continue in the background and clear the db of the reported missing container.

We still need to determine how this edge case would be invoked during an upgrade

[ddadlani]

We might also want to sync worker created containers and volumes with the ones in the database both ways, rather than cleaning up containers/volumes that we don't see from the DB. This has the added benefit of allowing us to remove the stalled state because we would simply re-sync the containers/volumes once the worker came back online.

One way to do this would be to have the worker be the source of truth for created containers and volumes, and have the db be the source of truth for creating/destroying.