cmd-compress: Use multithreading for xz#632
Conversation
jlebon
force-pushed
the
pr/compress-multi-threads
branch
from
6ba4d51 to
0f054e9
Compare
| in_k8s = re.search(r'.*kubepods.*', f.read()) | ||
| if in_k8s: | ||
| quota = get_cpu_param('cfs_quota_us') | ||
| # are k8s limits enforced? |
There was a problem hiding this comment.
Did you invent this code or take it from somewhere else?
There was a problem hiding this comment.
I re-read https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ and read https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu-time-9eff74d3161b.
It's also mostly the same thing the JDK HotSpot code does too though: https://hg.openjdk.java.net/jdk/jdk/file/7b671e6b0d5b/src/hotspot/os/linux/osContainer_linux.cpp#l595. Which, hmm yeah, should probably check that the period is > 0.
There was a problem hiding this comment.
OK, threw a link to that and added a check for the period.
I also found https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu really useful. Essentially though, I'm trying to avoid xz scheduling e.g. 20 threads because it sees 20 CPUs when really it'd be throttled at e.g. 200m.
xz compression is notoriously slow. Fortunately, it supports a multi-threaded mode. The compression ratio is affected by this, but not really in any significant way (see [1]). On k8s though, we have to make sure to respect any cgroup limits set on the CPU. (Though in CentOS right now, we run without any limits set at all, so this will make a huge dent in our build times). [1] https://yeah.nah.nz/misc/xz-thread/
jlebon
force-pushed
the
pr/compress-multi-threads
branch
from
0f054e9 to
82b9705
Compare
|
OK, thanks for the links - I think the context here will be really useful for someone reading this later. |
xz compression should be much faster now with: [1] coreos/coreos-assembler#632 So let's drop the hack and always use xz. For the developer use case, the MINIMAL parameter means there's only really the `qcow2` to compress.
xz compression should be much faster now with: [1] coreos/coreos-assembler#632 So let's drop the hack and always use xz. For the developer use case, the MINIMAL parameter means there's only really the `qcow2` to compress.
Otherwise, it defaults to `_SC_NPROCESSORS_ONLN` (via `%make_build` ->
`%_smp_mflags` -> `%_smp_build_ncpus` -> `%{getncpus}` ->
https://github.com/rpm-software-management/rpm/blob/48c0f28834eb377a54f27ee0b6950af7e6d537b8/rpmio/macro.c#L583).
And that's going to be wrong in Kubernetes because we're constrained via
cgroups.
The `%_smp_build_ncpus` macro allows overriding this logic via
`RPM_BUILD_NCPUS`.
See: coreos/coreos-ci#23
See: coreos/coreos-assembler#632
See: coreos/coreos-assembler#1287
Otherwise, it defaults to `_SC_NPROCESSORS_ONLN` (via `%make_build` ->
`%_smp_mflags` -> `%_smp_build_ncpus` -> `%{getncpus}` ->
https://github.com/rpm-software-management/rpm/blob/48c0f28834eb377a54f27ee0b6950af7e6d537b8/rpmio/macro.c#L583).
And that's going to be wrong in Kubernetes because we're constrained via
cgroups.
The `%_smp_build_ncpus` macro allows overriding this logic via
`RPM_BUILD_NCPUS`.
See: coreos/coreos-ci#23
See: coreos/coreos-assembler#632
See: coreos/coreos-assembler#1287
2 participants
xz compression is notoriously slow. Fortunately, it supports a
multi-threaded mode. The compression ratio is affected by this, but not
really in any significant way (see [1]).
On k8s though, we have to make sure to respect any cgroup limits set on
the CPU. (Though in CentOS right now, we run without any limits set at
all, so this will make a huge dent in our build times).
[1] https://yeah.nah.nz/misc/xz-thread/