`true` startup is slower than `gnutrue` startup

[danielzgtg]

Edit: I narrowed down the problem to uutils multi-call binary startup in general. Testing ./coreutils true vs gnutrue is a more minimal example.

---

Original report about ls and gnuls:

Someone subtly criticized Ubuntu's adoption of uutils, saying it was slower, despite me saying Rust is supposed to be faster.

For an ls of an empty directory, coreutils ls is much faster: perf stat -r 200 -e cycles,task-clock -- uu-ls /tmp/empty/ >/dev/null, on my i7-6700k Skylake at ~3.9GHz, uu-ls averaged about 2.0ms per invocation, Coreutils ls about 0.5ms. So about 1.5 ms less startup overhead, assuming the actual work takes no or equal time. (With a small directory like 10 files and a couple subdirs, they each take incrementally about the same amount more time, about 0.05 ms more, very roughly with noisy measurements, but repeating many times to warm up CPU to max MHz.) – Peter Cordes Commented

I tested an empty directory at /mnt.

Expected behavior

See GNU coreutils' ls' performance.

home@daniel-desktop3:~$ perf stat -r 200 -e cycles,task-clock -- /bin/gnuls /mnt >/dev/null

 Performance counter stats for '/bin/gnuls /mnt' (200 runs):

         2 162 945      cycles                           #    4,299 GHz                         ( +-  0,28% )
              0,50 msec task-clock                       #    0,624 CPUs utilized               ( +-  1,10% )

         0,0008064 +- 0,0000108 seconds time elapsed  ( +-  1,33% )

Actual behavior

uutils Ubuntu 25.10 is 120% slower. uutils 0f891cc is 108% slower. Peter Cordes said an unknown version was 300% slower.

home@daniel-desktop3:~$ /home/home/CLionProjects/uutils/target/release/coreutils ls --version
ls (uutils coreutils) 0.7.0
home@daniel-desktop3:~$ perf stat -r 200 -e cycles,task-clock -- /home/home/CLionProjects/uutils/target/release/coreutils ls /mnt >/dev/null

 Performance counter stats for '/home/home/CLionProjects/uutils/target/release/coreutils ls /mnt' (200 runs):

         5 770 027      cycles                           #    4,329 GHz                         ( +-  0,19% )
              1,33 msec task-clock                       #    0,751 CPUs utilized               ( +-  0,85% )

         0,0017746 +- 0,0000160 seconds time elapsed  ( +-  0,90% )
home@daniel-desktop3:~$ /bin/ls --version
/bin/ls (uutils coreutils) 0.2.2
home@daniel-desktop3:~$ perf stat -r 200 -e cycles,task-clock -- /bin/ls /mnt >/dev/null

 Performance counter stats for '/bin/ls /mnt' (200 runs):

         6 725 333      cycles                           #    4,635 GHz                         ( +-  0,17% )
              1,45 msec task-clock                       #    0,785 CPUs utilized               ( +-  0,77% )

         0,0018475 +- 0,0000146 seconds time elapsed  ( +-  0,79% )

[sylvestre]

Rust is supposed to be faster.

It is more complex than that. it depends how to program...

please run the benchmark with hyperfine

see
https://github.com/uutils/coreutils/blob/main/docs/src/performance.md#example-benchmark

[sylvestre]

also, benchmarking with an empty directories isn't interesting
you aren't benchmarking the code but just the startup

on a directory with more files (with -R)

 hyperfine --warmup 5 "/usr/bin/ls -R" "./target/release/coreutils ls -R"
Benchmark 1: /usr/bin/ls -R
  Time (mean ± σ):      97.4 ms ±   3.0 ms    [User: 60.7 ms, System: 36.6 ms]
  Range (min … max):    93.7 ms … 106.8 ms    29 runs

Benchmark 2: ./target/release/coreutils ls -R
  Time (mean ± σ):      94.8 ms ±   2.9 ms    [User: 51.4 ms, System: 43.2 ms]
  Range (min … max):    90.9 ms … 100.9 ms    32 runs

Summary
  ./target/release/coreutils ls -R ran
    1.03 ± 0.04 times faster than /usr/bin/ls -R

[danielzgtg]

please run the benchmark with hyperfine

home@daniel-desktop3:~/CLionProjects/uutils$ ls -1U /mnt | wc -l
0
home@daniel-desktop3:~/CLionProjects/uutils$ hyperfine --warmup 5 '/usr/bin/gnuls /mnt' '/usr/bin/ls /mnt' "./target/profiling/coreutils ls /mnt"
Benchmark 1: /usr/bin/gnuls /mnt
  Time (mean ± σ):     778.8 µs ± 106.1 µs    [User: 403.4 µs, System: 459.6 µs]
  Range (min … max):   603.4 µs … 1237.1 µs    2116 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
Benchmark 2: /usr/bin/ls /mnt
  Time (mean ± σ):       2.1 ms ±   0.2 ms    [User: 0.7 ms, System: 1.5 ms]
  Range (min … max):     1.7 ms …   2.7 ms    1054 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
Benchmark 3: ./target/profiling/coreutils ls /mnt
  Time (mean ± σ):       1.9 ms ±   0.2 ms    [User: 0.6 ms, System: 1.4 ms]
  Range (min … max):     1.5 ms …   3.3 ms    1128 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
Summary
  /usr/bin/gnuls /mnt ran
    2.39 ± 0.41 times faster than ./target/profiling/coreutils ls /mnt
    2.66 ± 0.44 times faster than /usr/bin/ls /mnt
home@daniel-desktop3:~/CLionProjects/uutils$ ls -1U /bin | wc -l
4834
home@daniel-desktop3:~/CLionProjects/uutils$ hyperfine --warmup 5 '/usr/bin/gnuls /bin' '/usr/bin/ls /bin' "./target/profiling/coreutils ls /bin"
Benchmark 1: /usr/bin/gnuls /bin
  Time (mean ± σ):       5.8 ms ±   0.4 ms    [User: 3.3 ms, System: 2.5 ms]
  Range (min … max):     5.2 ms …   6.9 ms    421 runs
 
Benchmark 2: /usr/bin/ls /bin
  Time (mean ± σ):       7.8 ms ±   0.6 ms    [User: 3.5 ms, System: 4.3 ms]
  Range (min … max):     7.0 ms …   9.6 ms    330 runs
 
Benchmark 3: ./target/profiling/coreutils ls /bin
  Time (mean ± σ):       6.6 ms ±   0.5 ms    [User: 2.6 ms, System: 4.0 ms]
  Range (min … max):     6.0 ms …   8.9 ms    377 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Summary
  /usr/bin/gnuls /bin ran
    1.14 ± 0.13 times faster than ./target/profiling/coreutils ls /bin
    1.36 ± 0.15 times faster than /usr/bin/ls /bin
home@daniel-desktop3:~/CLionProjects/uutils$ ls -1U /usr/share/man/man3 | wc -l
7846
home@daniel-desktop3:~/CLionProjects/uutils$ hyperfine --warmup 5 '/usr/bin/gnuls /usr/share/man/man3' '/usr/bin/ls /usr/share/man/man3' "./target/profiling/coreutils ls /usr/share/man/man3"
Benchmark 1: /usr/bin/gnuls /usr/share/man/man3
  Time (mean ± σ):      11.6 ms ±   0.6 ms    [User: 7.8 ms, System: 3.8 ms]
  Range (min … max):    11.0 ms …  13.7 ms    238 runs
 
Benchmark 2: /usr/bin/ls /usr/share/man/man3
  Time (mean ± σ):      11.8 ms ±   0.8 ms    [User: 5.6 ms, System: 6.2 ms]
  Range (min … max):    11.0 ms …  16.3 ms    242 runs
 
Benchmark 3: ./target/profiling/coreutils ls /usr/share/man/man3
  Time (mean ± σ):      10.1 ms ±   0.7 ms    [User: 4.0 ms, System: 6.1 ms]
  Range (min … max):     9.2 ms …  12.7 ms    270 runs
 
Summary
  ./target/profiling/coreutils ls /usr/share/man/man3 ran
    1.14 ± 0.10 times faster than /usr/bin/gnuls /usr/share/man/man3
    1.16 ± 0.12 times faster than /usr/bin/ls /usr/share/man/man3

you aren't benchmarking the code but just the startup

The benchmark found that the startup overhead is more important in a real-world scenario than the asymptotic time complexity of the code. Most directories have less than 100 children. /bin is already a big directory with 4834 childern but gnuls is still faster. Only unreasonably enormous directories like /usr/share/man/man3 with 7846 children benefit from "the code" being faster compared to the startup. And more people use find or fd instead of ls -R.

[sylvestre]

sure but we are only talking about 1.9ms or 2.1 ms in your first example

[danielzgtg]

The same problem happens with ./coreutils or ./coreutils true. Although bash has true built-in, this exemplifies the problem.

home@daniel-desktop3:~/CLionProjects/uutils/target/profiling$ hyperfine './coreutils ls'
Benchmark 1: ./coreutils ls
  Time (mean ± σ):       2.1 ms ±   0.2 ms    [User: 0.6 ms, System: 1.6 ms]
  Range (min … max):     1.6 ms …   2.7 ms    1004 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
home@daniel-desktop3:~/CLionProjects/uutils/target/profiling$ hyperfine './coreutils'
Benchmark 1: ./coreutils
  Time (mean ± σ):       1.6 ms ±   0.2 ms    [User: 0.5 ms, System: 1.2 ms]
  Range (min … max):     1.2 ms …   3.1 ms    1218 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
home@daniel-desktop3:~/CLionProjects/uutils/target/profiling$ hyperfine './coreutils true'
Benchmark 1: ./coreutils true
  Time (mean ± σ):       1.5 ms ±   0.2 ms    [User: 0.4 ms, System: 1.2 ms]
  Range (min … max):     1.2 ms …   2.0 ms    1281 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
 
home@daniel-desktop3:~/CLionProjects/uutils/target/profiling$ hyperfine '/bin/gnutrue'
Benchmark 1: /bin/gnutrue
  Time (mean ± σ):     484.7 µs ±  72.3 µs    [User: 301.3 µs, System: 229.7 µs]
  Range (min … max):   385.3 µs … 979.2 µs    2028 runs
 
  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: The first benchmarking run for this command was significantly slower than the rest (919.3 µs). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

I will attempt to profile this.

[oech3]

dupe of #9081 (which is closed as dupe of locale issue).

Rust has many some startup overheads to std (e.g. custom SIGSEGV). Nothing we can do at here. You should open an issse at upstream about fn main(){}.

[oech3]

It might be able to delay

coreutils/src/uucore/src/lib/lib.rs

Lines 193 to 208 in 309efd8

	// Preserve inherited SIGPIPE settings (e.g., from env --default-signal=PIPE)
	uucore::panic::preserve_inherited_sigpipe();
	// suppress extraneous error output for SIGPIPE failures/panics
	uucore::panic::mute_sigpipe_panic();
	locale::setup_localization(uucore::get_canonical_util_name(stringify!($util)))
	.unwrap_or_else(|err| {
	match err {
	uucore::locale::LocalizationError::ParseResource {
	error: err_msg,
	snippet,
	} => eprintln!("Localization parse error at {snippet}: {err_msg:?}"),
	other => eprintln!("Could not init the localization system: {other}"),
	}
	std::process::exit(99)
	});

at least for true/false.

[danielzgtg]

Today's profiling was fruitful. I achieved 16% performance gains at rust-lang/rust#153936 . It will be up to 8.91% or 0.2 ms for us:

I see we have panic = "abort" in Cargo.toml/[profile.release]. But panic = "immediate-abort" is nightly-only. Is there a way in Cargo.toml to create a new "release-small-nightly" profile that enables the options at https://github.com/johnthagen/min-sized-rust ? Otherwise, we could create a subdirectory.

A cargo new hello-world program takes 0.6938 ms. --release only optimizes 0.0365 ms. Forget GNU coreutils, and think about @oech3's fn main(){} example. That example provides /bin/true and takes 0.6468 ms debug and 0.6394 ms release. Our .coreutils true takes 1.5 ms. Therefore, there is an opportunity to save an extra 57% i.e. 0.8 ms.

hello-world is 4.2M debug and 430K release, so non-loaded debug symbol sections don't matter. But see bingrep /bin/ls "Dynamic Relas(41021):" vs bingrep /bin/gnuls "Dynamic Relas(233)". I would like to investigate why _dl_relocate_object takes 40.86% of the runtime for us, but only 6.79% for GNU Coreutils.

[oech3]

By the way, we support cargo build -p uu_true --release for individual bins. We can't make it defualt for binary size since -C prefer-dynamic breaks build for many utils and difficult to distribute.

[danielzgtg]

In my local branch, I got the multi-call binary without any arguments to be faster than 1 ms. I will also open a PR for fcntl when I wake up

[oech3]

Ref rust-lang/rust#153426