Document rake build:checksum by pboling · Pull Request #325 · rubygems/guides

pboling · 2022-10-27T22:51:01Z

Related to ruby/rubygems#5942 and ruby/rubygems#6022

Script uses a good regex, /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/, for pulling out a version match:
https://rubular.com/r/9QuDiGVjvlHOrt

deivid-rodriguez · 2023-06-16T13:59:46Z

security.md

+    rake build:checksum
+
+The checksums will be placed in the `checksums/` directory.  It is recommended to commit checksums so others can verify
+the authenticity of a release.


I would migrate the previous instructions to use the builtin task and that's it. Officially recommending it should be a different discussion.

Makes sense. I'll update soon.

Removed the "recommendation" implication. Also improved the script to what I am currently using as I'm not sure when I'll get back around to the rake task, since in its current form it isn't useful to me at all, and I have a deep todo list. I'll remove the script portion if preferred.

security.md

pboling · 2023-10-10T18:05:08Z

Very strangely Gem::Version.new(version) compared to another of the same isn't working for pre-release versions. Outside pre-release versions, the script works. 😖

ruby/rubygems#3086 (comment)

martinemde · 2023-10-10T22:35:49Z

security.md

+
+require "digest/sha2"
+
+VERSION_REGEX = /\d+\.\d+\.\d+([-.].+)*/.freeze


This is incorrect when applied to a filename since it will include .gem in the version number, making everything a pre-release.

Oooh, that's why it fails... interesting. I will see if I can find what rubygems uses for parsing gem names.

I knew it included the .gem, but it hadn't ever mattered until I had to deal with prereleases.

@martinemde Fixed!

pboling · 2023-10-11T06:14:18Z

# Final clause of Regex `(?=\.gem)` is a positive lookahead assertion
# See: https://learnbyexample.github.io/Ruby_Regexp/lookarounds.html#positive-lookarounds
# Used to pattern match against a gem package name, which always ends with .gem.
# The positive lookahead ensures it is present, and prevents it from being captured.
VERSION_REGEX = /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/.freeze

A good regex is hard to find. 🤣

pboling · 2023-10-12T17:22:42Z

security.md


 1) Create self-signed gem cert
-
+    ```shell


The enumerated list was not proper markdown, and after a few syntax changes, which should also allow language-specific code highlighting, it is now proper markdown.

pboling · 2023-10-12T17:23:13Z

security.md

 ### Reporting a security vulnerability with someone else's gem

-If you spot a security vulnerability in someone else's gem, then you
+If you spot a security vulnerability in someone else's gem, then your


Just a typo you => your

pboling · 2023-10-12T17:25:13Z

security.md

+# See: https://learnbyexample.github.io/Ruby_Regexp/lookarounds.html#positive-lookarounds
+# Used to pattern match against a gem package name, which always ends with .gem.
+# The positive lookahead ensures it is present, and prevents it from being captured.
+VERSION_REGEX = /((\d+\.\d+\.\d+)([-.][0-9A-Za-z-]+)*)(?=\.gem)/.freeze


This regex works perfectly.

https://rubular.com/r/9QuDiGVjvlHOrt

pboling · 2023-10-20T03:45:22Z

Unfortunate discovery. The process of running a checksum script or task outside of the rake release results in checksums that do not match the actual released gem.

❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

de3b77b9434b36157efde7dc3617dc037708a83560bd54367e0dc1d0c9492e66ddaa624f217ed6759599d6b8208405db2f2f1671bbf9c4c725b8f33ec6886909
❯ bin/checksums
Looking for: "pkg/*.gem"
Found: 4 gems; latest is kettle-soup-cover-1.0.2.gem
[ GEM: kettle-soup-cover-1.0.2.gem ]
[ VERSION: 1.0.2 ]
[ GEM PKG LOCATION: pkg/kettle-soup-cover-1.0.2.gem ]
[ CHECKSUM SHA-256: 677d0f4778e065be83d9e77b0514b64731cb4313fa2b085056e9ec45f92991a5 ]
[ CHECKSUM SHA-512: de3b77b9434b36157efde7dc3617dc037708a83560bd54367e0dc1d0c9492e66ddaa624f217ed6759599d6b8208405db2f2f1671bbf9c4c725b8f33ec6886909 ]
[ CHECKSUM SHA-256 PATH: checksums/kettle-soup-cover-1.0.2.gem.sha256 ]
[ CHECKSUM SHA-512 PATH: checksums/kettle-soup-cover-1.0.2.gem.sha512 ]

... Running ...

git add checksums/* && git commit -m "🔒️ Checksums for v1.0.2"

[main e62712c] 🔒️ Checksums for v1.0.2
 2 files changed, 2 insertions(+)
 create mode 100644 checksums/kettle-soup-cover-1.0.2.gem.sha256
 create mode 100644 checksums/kettle-soup-cover-1.0.2.gem.sha512
❯ bundle exec rake release
Enter PEM pass phrase:
kettle-soup-cover 1.0.2 built to pkg/kettle-soup-cover-1.0.2.gem.
Tagged v1.0.2.
Pushed git commits and release tag.
Pushing gem to https://rubygems.org...
You have enabled multi-factor authentication. Please enter OTP code.
Code:   077236
Successfully registered gem: kettle-soup-cover (1.0.2)
Pushed kettle-soup-cover 1.0.2 to rubygems.org
❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

80971e7cd27a4d22b8a8d93e869e7a2543a21455e5ae0700b0321268f555cc81381bec88b024017e9e74fa4a0cbfa4bbfe2ad1db2f5fc0b77259906fce15198d
❯ bundle exec rake build
Enter PEM pass phrase:
kettle-soup-cover 1.0.2 built to pkg/kettle-soup-cover-1.0.2.gem.
❯ ruby -rdigest/sha2 -e "puts Digest::SHA512.new.hexdigest(File.read('pkg/kettle-soup-cover-1.0.2.gem'))"

12354158c4a5c116aacb24cb16efd9b9a6240673c551042814b9f7e82d95de8d7cc87b5875a52e33bac660a5e182a581683af33cf7caef86930536d3006b2fe8

It seems that the only solution would be to run the checksums after release, but not running a new build of the gem package. The checksum commits would thus be after the release, and not in the git tag for the release. 😿

martinemde · 2023-10-20T04:38:02Z

Are you including the checksums files in the gem? Just a guess? (You would want to not do that.)

Is having the checksum in the tag even correct? How can a commit contain its own digest? Inception?

pboling · 2023-10-20T06:47:39Z

Are you including the checksums files in the gem?

Nope, I wondered the same thing, because if there were file changes between the gem builds, then they should have different checksums. Alas, no such luck.

Is having the checksum in the tag even correct?

I thought so, because the checksums are checksums of the built gem package. They have nothing to do with git.

How can a commit contain its own digest? Inception?

LOL, yeah it is a mind F8cker. It's really just that the gem package is wholly unrelated to the tracking of the project in git.

So the thing I don't understand is this - if the files are not changing, why does the checksum change everytime the gem is built? Either that's what is happening, or the packaged gem that results from rake build is fundamentally different than the packaged gem that results from rake release, and that lands somewhere between moderately and wildly disconcerting. 🤣 For a while I thought I was losing my mind, and then a pang of shame, realizing that every gem I have pushed, every checksum I have pushed does not match any of my releases, for the last year+. 🦺

I think I've found a bug. Because I assume that rake build and rake release should create identical packages, which would necessarily result in identical checksums.

pboling · 2023-10-20T06:51:10Z

To state as simply as possible:

Here's the repo I've been doing this evaluation in.

Somehow rake build and rake release create .gem packages that have different checksums. 😱

IMO, both tools are useful, but they must build identical packages...

segiddins · 2023-10-20T06:56:22Z

They do build identical packages, but with different timestamps unless you're setting the same SOURCE_DATE_EPOCH

pboling · 2023-10-20T06:58:43Z

Hero shit. ❤️ @segiddins. Thanks! I'll update to reflect that. 🔥

pboling · 2023-10-20T07:00:25Z

I think this should be prominently documented for both tasks, as for me, this is quite a violation of POLS. I'll work on PRs for that too. 🍔

Update

I haven't looked recently, so it if already is prominent, then I'm a victim of POLS assumptions. I may have never looked at the documentation for the tasks beyond the rake desc, because they seem extremely clearly named and obvious in function.

It seemed self-evident that both tasks should build an identical package, and I never considered timestamps, because that has never been on my radar... But this did cause me quite a problem, as I now have hundreds? of gems published with bad checksums, and bad release instructions. Since updated docs likely wouldn't have even helped me, I need to think about what a better solution might be.

deivid-rodriguez force-pushed the document-checksum branch from c1d04ac to dcc43c6 Compare June 16, 2023 13:55

deivid-rodriguez reviewed Jun 16, 2023

View reviewed changes

segiddins approved these changes Oct 10, 2023

View reviewed changes

pboling commented Oct 10, 2023

View reviewed changes

security.md Outdated Show resolved Hide resolved

martinemde reviewed Oct 10, 2023

View reviewed changes

🧑‍💻 Improved docs for checksums

2125037

📝 Markdown, syntax, and grammar fixes

de6d7fe

pboling commented Oct 12, 2023

View reviewed changes

📝 Explain usage of checksums

ae7a471


		require "digest/sha2"

		VERSION_REGEX = /\d+\.\d+\.\d+([-.].+)*/.freeze

Conversation

pboling commented Oct 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pboling commented Oct 10, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pboling commented Oct 11, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pboling commented Oct 20, 2023

Uh oh!

martinemde commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pboling commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pboling commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

segiddins commented Oct 20, 2023

Uh oh!

pboling commented Oct 20, 2023

Uh oh!

pboling commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pboling commented Oct 27, 2022 •

edited

Loading

martinemde commented Oct 20, 2023 •

edited

Loading

pboling commented Oct 20, 2023 •

edited

Loading

pboling commented Oct 20, 2023 •

edited

Loading

pboling commented Oct 20, 2023 •

edited

Loading