cloud-init sometimes fails on dpkg lock due to concurrent apt-daily-upgrade.service execution

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1711428

Launchpad details

affected_projects = ['apt (Ubuntu)']
assignee = None
assignee_name = None
date_closed = 2022-11-02T10:37:18.293399+00:00
date_created = 2017-08-17T17:55:30.356576+00:00
date_fix_committed = 2022-11-02T10:37:18.293399+00:00
date_fix_released = 2022-11-02T10:37:18.293399+00:00
id = 1711428
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1711428
milestone = None
owner = jbrowne
owner_name = Jim Browne
private = False
status = fix_released
submitter = jbrowne
submitter_name = Jim Browne
tags = []
duplicates = []

Launchpad user Jim Browne(jbrowne) wrote on 2017-08-17T17:55:30.356576+00:00

This is the same problem as https://bugs.launchpad.net/cloud-init/+bug/1693361, but with a different APT invoking service. In this case it is apt-daily-upgrade.service.

So, I guess add apt-daily-upgrade.service to the Before line in /lib/systemd/system/cloud-final.service along side apt-daily.service.

Or wait for an APT fix. Or retry APT commands when executing "packages:"

Reporting this to save someone else trouble, but I think we'll be rolling back to Trusty at this point. Hopefully the B LTS will have an alternative to systemd.

[ubuntu-server-builder]

Launchpad user Steve Langasek(vorlon) wrote on 2017-08-17T18:46:50.908727+00:00

The apt-daily-upgrade service already declares itself to be 'After=apt-daily.service'. So there should be strict sequencing here of cloud-final.service -> apt-daily.service -> apt-daily-upgrade.service. If that's not happening, this warrants analysis to understand why so that it can be fixed in the apt package directly rather than adding more relationships to the cloud-init unit.

[ubuntu-server-builder]

Launchpad user Jim Browne(jbrowne) wrote on 2017-08-17T19:20:20.845421+00:00

Hmmm. I confirm what you see.

I think this problem is arising because we have Packer run (due to LP#1693361):
systemctl disable apt-daily.service
systemctl disable apt-daily.timer

when building the AMI.

I guess if apt-daily.service is disabled the Before is not transitive from cloud-init to apt-daily-upgrade via apt-daily? Not seeing anything definitive in the docs after a quick glance.

[ubuntu-server-builder]

Launchpad user Julian Andres Klode(juliank) wrote on 2017-08-17T19:33:05.483896+00:00

Not that if B has After=A does not imply that the two don't run at the same time. If B is already running, A will be started as well. That's the reason why we added a lock file in apt:

if [ "$1" = "lock_is_held" ]; then
shift
else
# Maintain a lock on fd 3, so we can't run the script twice at the same
# time.
eval $(apt-config shell StateDir Dir::State/d)
exec 3>${StateDir}/daily_lock
if ! flock -w 3600 3; then
echo "E: Could not acquire lock" >&2
exit 1
fi

# We hold the lock.  Rerun this script as a child process, which
# can run without propagating an extra fd to all of its children.
"$0" lock_is_held "$@" 3>&-
exit $?

fi

[ubuntu-server-builder]

Launchpad user Steve Langasek(vorlon) wrote on 2017-08-17T19:49:13+00:00

On Thu, Aug 17, 2017 at 07:20:20PM -0000, Jim Browne wrote:

Hmmm. I confirm what you see.

I think this problem is arising because we have Packer run (due to LP#1693361):
systemctl disable apt-daily.service
systemctl disable apt-daily.timer

when building the AMI.

I guess if apt-daily.service is disabled the Before is not transitive
from cloud-init to apt-daily-upgrade via apt-daily? Not seeing anything
definitive in the docs after a quick glance.

Correct. Before/After only establishes ordering between enabled services
and is ignored for services that are not enabled.

Fixing this to enforce that apt-daily.service and apt-daily-upgrade.service
are both always enabled/disabled together would be done by adding
'Requires=apt-daily.service' to apt-daily-upgrade.service. But I think that
is too strict, because one may legitimately choose to disable only one of
the two services for some reason.

So I think this is something that you will want to fix in your image
mastering scripts.

[ubuntu-server-builder]

Launchpad user Jim Browne(jbrowne) wrote on 2017-08-17T19:59:00.014518+00:00

So I think this is something that you will want to fix in your image
mastering scripts.

I agree and am fine with this being marked INVALID.

However, is juliank's note about the implementation of After= a concern w.r.t. how LP#1693361 was resovled?

[ubuntu-server-builder]

Launchpad user Steve Langasek(vorlon) wrote on 2017-08-17T20:16:19+00:00

On Thu, Aug 17, 2017 at 07:59:00PM -0000, Jim Browne wrote:

However, is juliank's note about the implementation of After= a concern
w.r.t. how LP#1693361 was resovled?

I don't think so. He's correct that the ordering doesn't guarantee that we
don't hit a conflict if the units wrap around and collide in the opposite
direction. But I believe that locking code is part of the fix that landed.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2017-10-17T04:17:41.697212+00:00

[Expired for apt (Ubuntu) because there has been no activity for 60 days.]

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2022-11-02T10:37:11.746799+00:00

See #1034 and #1153