Improve sort buffer sizing heuristics and honor explicit --buffer-size by mattsu2020 · Pull Request #8833 · uutils/coreutils

mattsu2020 · 2025-10-06T12:01:53Z

Add automatic buffer-size heuristics (ported from commit a0e77d9). We now size external-sort chunks based on input file sizes and available memory, clamping to 512 KiB–128 MiB so we avoid both tiny buffers and risky overcommit on constrained systems.

Respect user-provided --buffer-size. Only automatically computed sizes are raised to the safety minimum; explicit values are left untouched, which keeps external sorting and --compress-program working even when users choose small buffers.

Performance Comparison (baseline vs. current)
Measurements come from hyperfine --warmup 3 --runs 10; values are means in milliseconds (lower is better).

Scenario	Baseline	Current	Delta	Speedup
ASCII 500k	17.04	14.59	-2.45	1.17×
Numeric 500k	36.89	37.29	+0.40	0.99×
ASCII 4M	112.47	107.52	-4.95	1.05×
ASCII 4M (-S 32M)	207.40	100.48	-106.92	2.06×
ASCII 16M	854.18	414.45	-439.73	2.06×
ASCII 16M (-S 512M)	838.05	426.85	-411.20	1.96×

github-actions · 2025-10-06T12:24:50Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-07T02:07:20Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)

github-actions · 2025-10-10T12:01:57Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-11T07:22:03Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

sylvestre · 2025-10-11T07:46:41Z

you have to refresh the fuzz/Cargo.lock file

github-actions · 2025-10-11T08:54:03Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

sylvestre · 2025-10-11T09:04:31Z

please follow this documentation for the performance work:
https://github.com/uutils/coreutils/blob/main/docs/src/performance.md

We would like to see hyperfine results and codspeed benchmark :)
thanks

.codespell.rc

src/uu/sort/src/sort.rs

sylvestre · 2025-10-11T09:09:08Z

src/uu/sort/src/sort.rs

+    quarter.max(max)
+}
+
+#[cfg(target_os = "linux")]


i was already not convinced in #8802
but
there is probably a better way than parsing /proc/meminfo

esp in the sort.rs code

Implemented it using a different method

I said it in the other pr but it does not belong to sort but uucore
And maybe we already have such functions

Is it total_physical_memory? Does it perform the same processing?
Is it acceptable to modify uucore to add available memory?

src/uu/sort/src/sort.rs

src/uu/sort/src/tmp_dir.rs

codspeed-hq · 2025-10-11T09:21:46Z

CodSpeed Performance Report

Merging #8833 will degrade performances by 3.49%

_{Comparing mattsu2020:sort_performan-ce (800f71d) with main (909da50)}

Summary

⚡ 16 improvements
❌ 2 regressions
✅ 88 untouched
⏩ 73 skipped¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
⚡	`sort_accented_data[500000]`	831.2 ms	361.9 ms	×2.3
⚡	`sort_ascii_only[500000]`	809.3 ms	353.1 ms	×2.3
⚡	`sort_case_insensitive[500000]`	427.8 ms	282.6 ms	+51.34%
⚡	`sort_case_sensitive[500000]`	337.7 ms	173.7 ms	+94.43%
❌	`sort_key_field[500000]`	699.8 ms	717 ms	-2.4%
⚡	`sort_mixed_data[500000]`	739.1 ms	327.6 ms	×2.3
❌	`sort_numeric[500000]`	1.2 s	1.2 s	-3.49%
⚡	`sort_reverse_locale[500000]`	830.1 ms	361 ms	×2.3
⚡	`sort_unique_locale[500000]`	1,150.9 ms	482.4 ms	×2.4
⚡	`sort_ascii_c_locale`	29.8 ms	22.9 ms	+29.82%
⚡	`sort_ascii_utf8_locale`	59.6 ms	46.2 ms	+28.98%
⚡	`sort_german_c_locale`	95.5 ms	39 ms	×2.4
⚡	`sort_german_locale`	95.5 ms	39 ms	×2.4
⚡	`sort_mixed_c_locale`	95.1 ms	38.6 ms	×2.5
⚡	`sort_mixed_utf8_locale`	95.1 ms	38.9 ms	×2.4
⚡	`sort_random_strings`	56.2 ms	31.8 ms	+76.81%
⚡	`sort_reverse_mixed`	94.2 ms	38.3 ms	×2.5
⚡	`sort_unique_mixed`	88.6 ms	39.2 ms	×2.3

73 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

sylvestre · 2025-10-11T09:23:41Z

impressive wins :)

github-actions · 2025-10-11T09:47:30Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-11T10:42:00Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

sylvestre · 2025-10-11T11:28:43Z

i am planning to merge #8746
because:

it arrived earlier
it does better than yours on the issue sort does not finish for large one line file #8583

github-actions · 2025-10-11T12:42:42Z

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-11T13:10:01Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-12T03:36:07Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-13T00:09:43Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

sylvestre · 2025-10-13T06:57:07Z

please try to avoid commit messages like "fix"
it doesn't help reviewers

sylvestre · 2025-10-13T07:05:31Z

src/uu/sort/src/sort.rs

 use fnv::FnvHasher;
 #[cfg(target_os = "linux")]
-use nix::libc::{RLIMIT_NOFILE, getrlimit, rlimit};
+use libc::{RLIMIT_NOFILE, rlimit};


it was more here. why do you move to libc ?
what was wrong with nix ?

Since I was using libc elsewhere, I thought it would be better to match that, but the original code is better.

github-actions · 2025-10-13T07:45:46Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

mattsu2020

・Add comment
・Change to Nix

src/uu/sort/src/sort.rs

src/uu/sort/src/tmp_dir.rs

src/uu/sort/src/sort.rs

src/uu/sort/src/tmp_dir.rs

src/uu/sort/src/sort.rs

mattsu2020 · 2025-10-13T06:59:27Z

src/uu/sort/src/sort.rs

+    quarter.max(max)
+}
+
+#[cfg(target_os = "linux")]


Implemented it using a different method

mattsu2020 · 2025-10-13T07:13:30Z

src/uu/sort/src/sort.rs

 use fnv::FnvHasher;
 #[cfg(target_os = "linux")]
-use nix::libc::{RLIMIT_NOFILE, getrlimit, rlimit};
+use libc::{RLIMIT_NOFILE, rlimit};


Since I was using libc elsewhere, I thought it would be better to match that, but the original code is better.

mattsu2020

Added comments to the source code.

src/uu/sort/src/ext_sort.rs

src/uu/sort/src/sort.rs

mattsu2020 · 2025-10-13T11:07:20Z

src/uu/sort/src/sort.rs

+    quarter.max(max)
+}
+
+#[cfg(target_os = "linux")]


Is it total_physical_memory? Does it perform the same processing?
Is it acceptable to modify uucore to add available memory?

src/uu/sort/src/sort.rs

github-actions · 2025-10-13T11:41:21Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

mattsu2020 · 2025-10-14T07:13:53Z

move memory functions to uucore

github-actions · 2025-10-16T04:22:20Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-16T10:02:59Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

src/uucore/src/lib/features/parser/parse_size.rs

sylvestre · 2025-10-16T10:05:58Z

src/uu/sort/src/ext_sort.rs

-    // Heuristically chosen: Dividing by 10 seems to keep our memory usage roughly
-    // around settings.buffer_size as a whole.
-    let buffer_size = settings.buffer_size / 10;
+    // Cap oversized buffer requests at 512MiB to avoid unnecessary allocations


please explain why 512 :)

cap automatic_buffer_size at 1 GiB so the reader and sorter never hold more than about 1 GiB of data at once; without that cap they could grab several gigabytes and waste memory.
Since it becomes 1GB when allocated simultaneously, I set it to 512MB.

github-actions · 2025-10-16T12:11:04Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-18T10:26:34Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

sylvestre · 2025-10-18T11:49:03Z

src/uucore/Cargo.toml

 tempfile = { workspace = true }

+[target.'cfg(target_os = "linux")'.dependencies]
+procfs = "0.18"


Please use the one from workspace

#8968
fix PR

sylvestre · 2025-10-18T11:49:40Z

Looks great.
Now, I don't want the merge these 54 commits or squash into one.
Could you please clean it up? First, uucore, then memory functions and the rest

mattsu2020 · 2025-10-18T12:36:38Z

Looks great. Now, I don't want the merge these 54 commits or squash into one. Could you please clean it up? First, uucore, then memory functions and the rest

Fitst uucore change
#8951

github-actions · 2025-10-20T08:44:27Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

mattsu2020 · 2025-10-20T08:46:53Z

Looks great. Now, I don't want the merge these 54 commits or squash into one. Could you please clean it up? First, uucore, then memory functions and the rest

Fitst uucore change #8951
Second memory function change
#8959

sylvestre · 2025-10-20T08:50:21Z

src/uu/sort/src/sort.rs

+    }
+}
+
+fn file_size_hint(files: &[OsString]) -> Option<usize> {


please document this
and maybe it would make sense to move these memory functions into a specific file

sylvestre · 2025-10-20T08:51:21Z

src/uu/sort/src/sort.rs

+fn physical_memory_bytes() -> Option<u128> {
+    #[cfg(all(target_family = "unix", not(target_os = "redox")))]
+    {
+        let pages = unsafe { libc::sysconf(libc::_SC_PHYS_PAGES) };


can you use nix instead of libc here?
it will probably remove unsafe

coreutils/src/uu/sort/Cargo.toml

Line 40 in 8e7c6b0

[target.'cfg(target_os = "linux")'.dependencies]

Is it okay to modify this part too?
Since it depends on Linux.

maybe ? please give it a try :)

#8959 (comment)
In some environments, it seems the environment variables are missing.

github-actions · 2025-10-21T06:55:47Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-21T09:03:57Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

github-actions · 2025-10-22T05:22:55Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

- move heuristics into a new buffer_hint module and default to automatic sizing when the buffer flag is absent - tune chunk and external sort buffers to avoid runaway allocations - add fast lexicographic and ASCII case-insensitive comparisons for the default mode - refresh spell-check and dependency metadata for the new code

- keep the latest path/lock pair in a shared registry so SIGINT always cleans the active directory - guard handler installation with an atomic flag and reset state when the wrapper is dropped

github-actions · 2025-10-22T06:09:28Z

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

Remove Linux-specific dynamic adjustment based on file descriptors and use a fixed batch size of 64 for consistency across platforms.

github-actions · 2025-10-23T10:48:21Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)