init: Add --transient flag for ephemeral builds#2889
Conversation
This pairs with coreos/coreos-assembler#2889 If fsync is disabled on our target repository, take that as a hint we should not care about durability for our cache repositories either.
I kind of feel like disabling |
Yes, in theory. But...the hairy thing is correctly detecting that. A perfect example here is today, if one does In that case, you may end up with zero-sized or truncated or otherwise corrupted cached objects in ostree does this thing with using the boot-id but it's still not perfect. I mean, I agree with you obviously, it's just one of those things where the last 10% corner cases is 90% of the work. So for now...I think we need to just make this opt-in for CI. |
|
Good point. We could "big hammer" this and just force clean caches if uptime is newer than last time the cache was touched. Or warn the user. |
|
gangplank tests are being flaky (same thing observed on #2892); restarted test |
|
OK I pushed more changes here, most notably a bit that avoids the need for coreos/rpm-ostree#3719 in the virt case (most important one here). |
jlebon
left a comment
jlebon
left a comment
There was a problem hiding this comment.
We'll want to make sure to update coreos-ci-lib and the pipelines to make use of this too.
We go to a lot of effort to create various caches of builds, and this is very useful for local incremental development to avoid re-downloading RPMs and rewriting data. It is useless for CI builds that discard everything when they're done. Add a `cosa init --transient` flag that for now just disables `fsync()` on `tmp/repo`. There's more we can do here in the future, for example we need to propagate this too into `cache/repo-build` and `cache/pkgcache-repo`, and actually in general we don't need the pkgcache repo at all here.
To make it clearer that things worked.
Followup to the addition of `--transient` for CI flows. If we had coreos/rpm-ostree#3719 this would mostly be unnecessary, but today in CI and prod builds, ostree is invoking `fsync()` inside the supermin VM, and then qemu in the container is going to `fsync()` all changes down to the host system (overlay)fs. For transient flows, switch to `cache=unsafe` so we stop doing that which should greatly help speed.
So we test it, and so we also gain the speed benefits.
Opt into the faster infrastructure from coreos/coreos-assembler#2889 for CI builds where we do not maintain any state across builds.
4 participants
init: Add
--transientflag for ephemeral buildsWe go to a lot of effort to create various caches of builds, and this
is very useful for local incremental development to avoid re-downloading
RPMs and rewriting data.
Conversely, it is useless for CI builds that discard everything when they're done, and actually slows things down.
Add a
cosa init --transientflag that for now just disablesfsync()on
tmp/repo.There's more we can do here in the future, for example we need to
propagate this too into
cache/repo-buildandcache/pkgcache-repo,and actually in general we don't need the pkgcache repo at all here.
init: Add a success message
To make it clearer that things worked.