cksum: implement check (Closes: #5705) by sylvestre · Pull Request #6390 · uutils/coreutils

sylvestre · 2024-05-09T20:55:11Z

No description provided.

github-actions · 2024-05-10T07:49:23Z

GNU testsuite comparison:

GNU test failed: tests/cksum/cksum-a. tests/cksum/cksum-a is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/cksum/cksum-c is no longer failing!

github-actions · 2024-05-10T08:26:29Z

GNU testsuite comparison:

GNU test failed: tests/cksum/cksum-a. tests/cksum/cksum-a is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/cksum/cksum-c is no longer failing!
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

sylvestre · 2024-05-11T07:19:46Z

should fail but getting close

github-actions · 2024-05-11T07:47:11Z

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

github-actions · 2024-05-12T09:17:59Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!

sylvestre · 2024-05-12T11:32:39Z

coreutils/coreutils@b5ce9fb I contributed this upstream as they were missing

github-actions · 2024-05-12T15:25:07Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!

github-actions · 2024-05-12T19:39:40Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

github-actions · 2024-05-12T20:09:57Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

github-actions · 2024-05-12T21:15:36Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

BenWiederhake

I found a few places where we produce the wrong result, and even a panic :)

I'm weirdly proud of finding so much. Clearly it's already good code, it passes the GNU tests and even some that you created for them, and yet there are a lot of interesting edge cases! :D

BenWiederhake · 2024-05-12T23:21:11Z

+            io::ErrorKind::Other,
+            "no properly formatted checksum lines found",


For some reason, this does not always trigger for me:

$ ../gnu/src/cksum -a blake2b README.md | tee foo.sums BLAKE2b (README.md) = c0a230cfdbbb686f91071119c915a19df28cb2e4fc575f19d5c6b81bf4e3ca2e83e6df49f14d8b89ece14eb4090be435a7af6fed2c981774e076caa8fe62b04c $ ../gnu/src/cksum -a sha1 -c foo.sums ../gnu/src/cksum: foo.sums: no properly formatted checksum lines found [$? = 1] $ cargo run -q cksum -a sha1 -c foo.sums cksum: WARNING: 1 line is improperly formatted

I suspect that this has to do with -a sha1 being interpreted as "every line MUST use the specified algorithm, even if tagged otherwise" or something?

This bug still exists. We definitely shouldn't accept files if they aren't correct.

github-actions · 2024-05-14T17:52:04Z

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

github-actions · 2024-05-14T21:56:45Z

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

github-actions · 2024-05-15T10:16:22Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!

sylvestre · 2024-05-15T20:01:33Z

@BenWiederhake I added tests on our side (ignored) for stuff that we don't support yet (and probably quite niche)
As GNU doesn't have tests either, i proposed that we land this PR + i will contribute these tests upstream.
thanks!

BenWiederhake

Most of the problems I addressed are indeed fixed, and I mostly agree that it's a good idea to move most of the remainder for later. Hooray!

There are two exceptions that I think should be fixed in this PR:

The new-ish issue SHA4 (README.md) = 00000000, see above
The bit-rounding-down issue, which causes us to accept dubious and obviously-wrong hashes. This should be just a simple div-mod, right?

Things that should be fixed eventually, but perhaps not right now:

"accept files with explicit algo but algos mismatch" details
inconsistent behavior caused by the inclusion of b in the algo-name regex:

$ echo 'bSD (README.md) = 0000' > foo.sums; cargo run -q cksum -c foo.sums
README.md: FAILED
cksum: WARNING: 1 computed checksum did NOT match
[$? = 1]
$ echo 'BsD (README.md) = 0000' > foo.sums; cargo run -q cksum -c foo.sums
cksum: foo.sums: no properly formatted checksum lines found
cksum: WARNING: 1 line is improperly formatted
[$? = 1]

sylvestre · 2024-05-17T20:10:52Z

Done :)

inconsistent behavior caused by the inclusion of b in the algo-name regex:
fixed too

github-actions · 2024-05-17T20:38:11Z

GNU testsuite comparison:

Congrats! The gnu test tests/cksum/cksum-c is no longer failing!

BenWiederhake

No more crashes, no more false-positives in the checking, and I couldn't find any new edge cases in the new code. Yay! This is ready to merge in my eyes.

BenWiederhake · 2024-05-18T21:46:47Z

Remaining failures seem to be the usual (#6275, #6333, and that new "year 1677" thing on mac), so it's as green as it can be, currently.

sylvestre marked this pull request as ready for review May 12, 2024 11:17

sylvestre requested review from BenWiederhake and cakebaker May 12, 2024 11:17

cakebaker reviewed May 12, 2024

View reviewed changes

Comment thread src/uu/cksum/Cargo.toml Outdated

cakebaker reviewed May 12, 2024

View reviewed changes

Comment thread tests/by-util/test_cksum.rs Outdated

cakebaker reviewed May 12, 2024

View reviewed changes

Comment thread tests/by-util/test_cksum.rs Outdated

cakebaker reviewed May 12, 2024

View reviewed changes

Comment thread tests/by-util/test_cksum.rs Outdated

sylvestre added 3 commits May 12, 2024 21:20

cksum: move the length mgmt into a function + add tests

7e33650

cksum: use the qualified import to make its provenance clear

5f0b48e

cksum: implement check & strict

81500ae

sylvestre requested a review from cakebaker May 12, 2024 21:04

BenWiederhake requested changes May 13, 2024

View reviewed changes

sylvestre added 7 commits May 14, 2024 23:21

uucore: create a new function to manage the warning/error display

0b04bca

chksum: add support of --check --algorithm=xxx

1aec8b4

cksum/blake2b, when length=512, don't add it in the line

8ddb213

cksum/blake2b: improve the error management

843275a

cksum: various improvements/fixes

bbd80e4

cksum: handle the empty lines (and just spaces)

07d579c

cksum: also show the error on stdout

115b203

cksum: add failing tests - to be fixed

58c2b5d

cksum: also handle the case when the line start with ' \'

9c52ca8

BenWiederhake requested changes May 15, 2024

View reviewed changes

sylvestre added 3 commits May 17, 2024 20:21

cksum: fails in case of unknown algo

1b78102

cksum: when length/bits can't be divided by 8, generate an error

db58d2b

cksum: improve the blake2b detection algo corner case

514d810

sylvestre requested a review from BenWiederhake May 18, 2024 06:02

sylvestre changed the title ~~cksum: implement check~~ cksum: implement check (Closes: #5705) May 18, 2024

BenWiederhake approved these changes May 18, 2024

View reviewed changes

BenWiederhake merged commit b718f95 into uutils:main May 18, 2024

BenWiederhake mentioned this pull request May 20, 2024

flaky CI macos: stat reports year 1677 #6419

Closed

sylvestre deleted the cksum-check branch May 26, 2024 16:54

BrewTestBot mentioned this pull request Jun 23, 2024

uutils-coreutils 0.0.27 Homebrew/homebrew-core#175400

Merged

moonfruit mentioned this pull request Jun 24, 2024

uutils-selected 0.0.27 moonfruit/homebrew-tap#114

Merged

		io::ErrorKind::Other,
		"no properly formatted checksum lines found",

Uh oh!

Conversation

sylvestre commented May 9, 2024

Uh oh!

github-actions Bot commented May 10, 2024

Uh oh!

github-actions Bot commented May 10, 2024

Uh oh!

sylvestre commented May 11, 2024

Uh oh!

github-actions Bot commented May 11, 2024

Uh oh!

github-actions Bot commented May 12, 2024

Uh oh!

sylvestre commented May 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 12, 2024

Uh oh!

github-actions Bot commented May 12, 2024

Uh oh!

github-actions Bot commented May 12, 2024

Uh oh!

github-actions Bot commented May 12, 2024

Uh oh!

BenWiederhake left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

BenWiederhake May 12, 2024

Choose a reason for hiding this comment

Uh oh!

BenWiederhake May 15, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 14, 2024

Uh oh!

github-actions Bot commented May 14, 2024

Uh oh!

github-actions Bot commented May 15, 2024

Uh oh!

sylvestre commented May 15, 2024

Uh oh!

BenWiederhake left a comment

Choose a reason for hiding this comment

Uh oh!

sylvestre commented May 17, 2024

Uh oh!

github-actions Bot commented May 17, 2024

Uh oh!

BenWiederhake left a comment

Choose a reason for hiding this comment

Uh oh!

BenWiederhake commented May 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants