stdenv: set SOURCE_DATE_EPOCH to a value python supports #89794

Mic92 · 2020-06-08T10:56:57Z

in nix-shell this value breaks the build because python's
packaging refuses to build timestamps that date before 1980.

Motivation for this change

I am surprised this has been not done before... Tell me if I miss anything here.
cc @FRidh @jonringer

Things done

in nix-shell this value breaks the build because python's packaging refuses to build timestamps that date before 1980.

Mic92 · 2020-06-08T10:58:04Z

If there are no objections I would also fix the python docs before merging this.

jonringer · 2020-06-08T18:28:27Z

This is handled for python packages using the ensureNewerSourcesForZipFilesHook, should be used by default if buildPython{Packages,Application} is used as the mkDerivation helper.

jonringer · 2020-06-08T18:30:30Z

I think this has pretty big ramifications from a philosophical / purist perspective.

cc @edolstra

EDIT: also, this opens of the door to having to support any new archive format which may or may not support a given date range.

Mic92 · 2020-06-08T21:37:33Z

This is handled for python packages using the ensureNewerSourcesForZipFilesHook, should be used by default if buildPython{Packages,Application} is used as the mkDerivation helper.

Well. it is sort of annoying so and it is not added when you add python to buildInputs.

Mic92 · 2020-06-08T21:39:20Z

I think this has pretty big ramifications from a philosophical / purist perspective.

I don't get this point. Please backup with technical reasons.

Mic92 · 2020-06-08T21:42:11Z

EDIT: also, this opens of the door to having to support any new archive format which may or may not support a given date range.

I don't see any reason not to change it again if we have good reason to. It does not comes with any cost if time it with another stdenv rebuild.

jonringer · 2020-06-08T23:27:33Z

I don't get this point. Please backup with technical reasons.

I'm definitely not the best to comment on this. cc @FRidh

Trust me, I think it's annoying as well. I've just gotten used to including the hook to all my environments.

FRidh · 2020-06-13T07:56:02Z

See earlier #60446

Mic92 · 2020-06-14T07:20:16Z

I don't agree that we have to switch nix store dates. In a nix-shell it is enough to have SOURCE_DATE_EPOCH with the value given here set for python wheels to be build successfully. If that is all arguments against changing this value I would like to proceed to merge it.

FRidh · 2020-06-14T07:36:42Z

If Python packaging using Nix needs it, then it needs to propagate the hook. Just make sure you propagate from the right tools.

Mic92 · 2020-06-14T08:44:55Z

If Python packaging using Nix needs it, then it needs to propagate the hook. Just make sure you propagate from the right tools.

So should we added it to python's setup hook?

FRidh · 2020-06-14T08:51:07Z

The interpreter does not do anything with wheels. The logical place would be the tools consuming or producing the wheels. When again does the issue occur, when building or installing wheels? If I am correct it is when building wheels. Thus the PEP 517 backends should propagate the hook. setuptools, flit-core, mesonpep517, poetry.

Mic92 · 2020-06-14T08:54:58Z

The interpreter does not do anything with wheels. The logical place would be the tools consuming or producing the wheels. When again does the issue occur, when building or installing wheels? If I am correct it is when building wheels. Thus the PEP 517 backends should propagate the hook. setuptools, flit-core, mesonpep517, poetry.

Still this is bad UX. If someone includes python in buildInputs and uses venv to create a virtual environment, they just expect it to work and not having to include additional hooks. So why not set SOURCE_DATE_EPOCH to a reasonable value in the first place. There has been no technical reason given against it so far.

FRidh · 2020-06-14T09:04:03Z

That's just because you use nix-shell, which gives an environment corresponding to a build environment. Using it for different purposes is not what it is meant for. Yes, its a convenient tool to use, we do it all the time, but you should be aware of its limitations.

Mic92 · 2020-06-14T09:54:36Z

That's just because you use nix-shell, which gives an environment corresponding to a build environment. Using it for different purposes is not what it is meant for. Yes, its a convenient tool to use, we do it all the time, but you should be aware of its limitations.

nix-shell is the only time available at the time for build environments in nix. You cannot even use a C compiler properly without it because there is no other way to build include/library paths. Still you have not answered the question why we cannot change this value.

FRidh · 2020-06-14T09:58:42Z

I don't know if it can or cannot be changed, and suggest you ask around further in the community, considering this is the stdenv.

timokau · 2020-06-14T10:20:23Z

I agree with @Mic92. The python issue is a major annoyance with a non-trivial fix. We can work around it with little to no cost. The fix isn't perfect, but strictly better than the status quo.

Set the floor of SOURCE_DATE_EPOCH to 1980, as ZIP file format does not support timestamps prior to 1980, meaning zip, jar, egg, wheel all fail to be created or used. Without this fix, if one removes the hand override of SOURCE_DATE_EPOCH from py3-userpath, the build fails: ```console melange build py3-userpath.yaml --arch x86_64 --pipeline-dir ./pipelines/ -k https://packages.wolfi.dev/os/wolfi-signing.rsa.pub -r https://packages.wolfi.dev/os --source-dir ./py3-userpath/ ... 2024/09/27 14:55:33 INFO * Building wheel... name="Python Build" 2024/09/27 14:55:33 WARN Traceback (most recent call last): name="Python Build" 2024/09/27 14:55:33 WARN File "/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module> name="Python Build" 2024/09/27 14:55:33 WARN main() name="Python Build" 2024/09/27 14:55:33 WARN File "/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main name="Python Build" 2024/09/27 14:55:33 WARN json_out["return_val"] = hook(**hook_input["kwargs"]) name="Python Build" 2024/09/27 14:55:33 WARN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ name="Python Build" 2024/09/27 14:55:33 WARN File "/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 271, in build_wheel name="Python Build" 2024/09/27 14:55:33 WARN return _build_backend().build_wheel( name="Python Build" 2024/09/27 14:55:33 WARN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ name="Python Build" 2024/09/27 14:55:33 WARN File "/tmp/build-env-u9wepgmk/lib/python3.12/site-packages/hatchling/build.py", line 58, in build_wheel name="Python Build" 2024/09/27 14:55:33 WARN return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard']))) name="Python Build" 2024/09/27 14:55:33 WARN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ name="Python Build" 2024/09/27 14:55:33 WARN File "/tmp/build-env-u9wepgmk/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 155, in build name="Python Build" 2024/09/27 14:55:33 WARN artifact = version_api[version](directory, **build_data) name="Python Build" 2024/09/27 14:55:33 WARN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ name="Python Build" 2024/09/27 14:55:33 WARN File "/tmp/build-env-u9wepgmk/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 476, in build_standard name="Python Build" 2024/09/27 14:55:33 WARN record = archive.add_file(included_file) name="Python Build" 2024/09/27 14:55:33 WARN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ name="Python Build" 2024/09/27 14:55:33 WARN File "/tmp/build-env-u9wepgmk/lib/python3.12/site-packages/hatchling/builders/wheel.py", line 98, in add_file name="Python Build" 2024/09/27 14:55:33 WARN zip_info = zipfile.ZipInfo(relative_path, cast(TIME_TUPLE, self.time_tuple)) name="Python Build" 2024/09/27 14:55:33 WARN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ name="Python Build" 2024/09/27 14:55:33 WARN File "/usr/lib/python3.12/zipfile/__init__.py", line 398, in __init__ name="Python Build" 2024/09/27 14:55:33 WARN raise ValueError('ZIP does not support timestamps before 1980') name="Python Build" 2024/09/27 14:55:33 WARN ValueError: ZIP does not support timestamps before 1980 name="Python Build" 2024/09/27 14:55:33 INFO name="Python Build" 2024/09/27 14:55:33 INFO ERROR Backend subprocess exited when trying to invoke build_wheel name="Python Build" 2024/09/27 14:55:33 ERRO ERROR: failed to build package. the build environment has been preserved: 2024/09/27 14:55:33 INFO workspace dir: /tmp/melange-workspace-1020228742 2024/09/27 14:55:33 INFO guest dir: /tmp/melange-guest-609671933 2024/09/27 14:55:33 ERRO failed to build package: unable to run package py3-userpath pipeline: unable to run pipeline: exit status 1 ``` With this fix inplace it just builds correctly. This will remove many custom overrides of SOURCE_DATE_EPOCH in wolfi-dev/os, and prevent any developer hitting such pitfall. This is similar to what was done in NixOS, Debian, Python and probably other projects. In Wolfi this will fix building brand new packages, and building things directly with melange/wolfictl outside of pipelines and makefiles. See for example NixOS/nixpkgs#89794

stdenv: set SOURCE_DATE_EPOCH to a value python supports

bc4927a

in nix-shell this value breaks the build because python's packaging refuses to build timestamps that date before 1980.

Mic92 requested review from Ericson2314 and matthewbauer as code owners June 8, 2020 10:56

Mic92 requested review from adisbladis, edolstra and vcunat June 14, 2020 10:12

timokau approved these changes Jun 14, 2020

View reviewed changes

Mic92 merged commit f6a30fc into NixOS:staging Aug 1, 2020

Mic92 deleted the source-date-epoch branch August 1, 2020 09:08

abathur mentioned this pull request Sep 22, 2021

resholve: 0.5.1 -> 0.6.0, refactor, +binlore #138080

Merged

12 tasks

FRidh mentioned this pull request Jan 16, 2022

Change default date of nix store to increase compatibility? #60446

Closed

xnox mentioned this pull request Sep 27, 2024

build: valid SOURCE_DATE_EPOCH chainguard-dev/melange#1520

Open

ctrlaltf2 mentioned this pull request May 19, 2025

ZipFile tests fail in negative timezone offsets on reproducible build environments python/cpython#134261

Open

Uh oh!

stdenv: set SOURCE_DATE_EPOCH to a value python supports #89794

stdenv: set SOURCE_DATE_EPOCH to a value python supports #89794

Uh oh!

Conversation

Mic92 commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for this change

Things done

Uh oh!

Mic92 commented Jun 8, 2020

Uh oh!

jonringer commented Jun 8, 2020

Uh oh!

jonringer commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mic92 commented Jun 8, 2020

Uh oh!

Mic92 commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mic92 commented Jun 8, 2020

Uh oh!

jonringer commented Jun 8, 2020

Uh oh!

FRidh commented Jun 13, 2020

Uh oh!

Mic92 commented Jun 14, 2020

Uh oh!

FRidh commented Jun 14, 2020

Uh oh!

Mic92 commented Jun 14, 2020

Uh oh!

FRidh commented Jun 14, 2020

Uh oh!

Mic92 commented Jun 14, 2020

Uh oh!

FRidh commented Jun 14, 2020

Uh oh!

Mic92 commented Jun 14, 2020

Uh oh!

FRidh commented Jun 14, 2020

Uh oh!

timokau commented Jun 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Mic92 commented Jun 8, 2020 •

edited

Loading

jonringer commented Jun 8, 2020 •

edited

Loading

Mic92 commented Jun 8, 2020 •

edited

Loading