From e6a3034493393a650c007713bcdff2756a8c296c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul=20=28ACSONE=29?= Date: Mon, 22 Apr 2019 00:16:57 +0200 Subject: [PATCH 1/7] Recording the Direct URL Origin of installed distributions --- pep-9999.rst | 303 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 303 insertions(+) create mode 100644 pep-9999.rst diff --git a/pep-9999.rst b/pep-9999.rst new file mode 100644 index 00000000000..6d76067e5e4 --- /dev/null +++ b/pep-9999.rst @@ -0,0 +1,303 @@ +PEP: 9999 +Title: Recording the source URL of an installed distribution +Author: Stéphane Bidoul +Sponsor: ??? +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 21-Apr-2019 +Post-History: +Discussion-To: https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535 + +Abstract +======== + +This PEP specifies a ``source_url.json`` file in the Database of Installed +Python Distributions PEP 376. The purpose of this file is to record, in an +extensible manner, the origin of distributions that were installed +from direct URL references (as opposed to distributions that were installed +from a package index). + +The main motivation of this PEP is allowing tools attempting to "freeze" the +state of a python environment to work in a broader range of situations. + +Motivation +========== + +This PEP originated from the need to implement `pip issue #609`_: +i.e. improving the behavior of ``pip freeze`` in presence of distributions +installed from direct URL references. It follows a +`thread on discuss.python.org`_ about the best course of action to implement +it. + +Installation from URL +--------------------- + +Python installers such as pip are capable of downloading and installing +distributions from package indexes. They are also capable of downloading +and installing source code from requirements specifying arbitrary URLs of +source archives and Version Control Systems (VCS) repositories, +as standardized in `PEP440 Direct References`_. + +In other words two relevant installation modes exist. + +1. the package to install is specified as a name and version specifier: + + In this case, the installer looks in a package index (or optionally + using --find-links in the case of pip) to find the distribution to install. + +2. The package to install is specified as a direct URL reference: + + In this case, the installer downloads whatever is specified by the URL + (typically a wheel, a source archive or a VCS repository) and installs it. + + In this mode, installers typically download the source code in a + temporary directory, invoke the PEP 517 build backend to produce a wheel + if needed, install the wheel, and delete the temporary directory. + + After installation, no trace of the URL the user requested to download the + package is left on the user system. + +Freezing an environment +----------------------- + +Pip also sports a command named ``pip freeze`` which examines the Database of +Installed Python Distributions to generate a list of requirements. The main +goal of this command is to help users generating a list of requirements that +will later allow the re-installation the same environment with the highest +possible fidelity. + +The ``pip freeze`` command outputs a ``name==version`` line for each installed +distribution (except for editable installs). To achieve the goal of +reinstalling the same environment, this requires the (name, version) +tuple to refer to an immutable version of the +distribution. The immutability is guaranteed by package indexes +such as Warehouse. The package index to use is typically known from +environmental or command line parameters of the installer. + +This freeze mechanism therefore works fine for installation mode 1 (i.e. +when the package to install was specified as a name plus version specifier). + +For installation mode 2, i.e. when the package to install was specified as a +direct URL reference, the ``name==version`` tuple is obviously not sufficient +to reinstall the same distribution and users of the freeze command expect it +to output the URL that was originally requested. + +The reasoning above is equally applicable to tools, other than ``pip freeze``, +that would attempt to generate a ``Pipfile.lock`` or any other similar format +from the Database of Installed Python Distributions. Unless specified +otherwise, "freeze" is used in this document as a generic term for such +an operation. + +The importance of installing from (VCS) URLs for application integrators +------------------------------------------------------------------------ + +For an application integrator, it is important to be able to reliably install +and freeze unreleased version of python distributions. +For instance when a developer needs to deploy an unreleased patched version +of a dependency, it is common to install the dependency directly from a VCS +branch that has the patch, while waiting for the maintainer to release an +updated version. + +In such cases, it is important for "freeze" to pin the exact VCS +reference (commit-hash if available) that was installed, in order to create +reproducible builds with the highest possible fidelity. + +Note about "editable" installs +------------------------------ + +The editable installation mode of pip roughly lets a user insert a +local directory in sys.path for development purpose. This mode is somewhat +abused to work around the fact that a non editable install from a VCS URL +loses trace of the origin after installation. +Indeed editable installs implicitly record the VCS origin in the checkout +directory, so the information can be recovered when running "freeze". + +The use of this workaround; although useful, is fragile, creates confusion +about the purpose of the editable mode, and works only when the distribution +can be installed with setuptools (i.e. it is not usable with other pep517 +build backends). + +For the sake of clarity, it is important to note that this PEP is otherwise +unrelated to editable installs. + +Rationale +========= + +This PEP specifies a new ``source_url.json`` metadata file in the .dist-info +directory of an installed distribution. + +The fields specified are sufficient to reproduce the source archive and `VCS +URLs supported by pip`_. They are also sufficient to reproduce +`PEP440 Direct References`_, as well as `Pipfile and Pipfile.lock`_ entries. + +Since at least the above 3 different way to encode the information exist, +this PEP uses a key-value format, to not make any assumption on how a source +URL must ultimately be encoded in a requirement or lockfile. See also +the `Alternatives`_ section below for more discussion about this choice. + +Information has been taken from Ruby's bundler manual to verify it has similar +capabilities and inform the selection and naming of fields in this +specifications. + +The json format allows for the addition of additional fields in the future. + +Specification +============= + +This PEP specifies a ``source_url.json`` file in the ``.dist-info`` directory +of an installed distribution. + +This file MUST be created by installers when installing a distribution +from a requirement specifying a direct URL reference (including a VCS URL +in *non*-editable mode). + +This file MUST NOT be created when installing a distribution from an other +type of requirement (i.e. name plus version specifier, or URL in editable mode). + +If present, it MUST contain at least one field with name ``url``. + +``url`` MUST be stripped of any sensitive authentication information, +for security reasons. The user:password section of the URL MAY however +be composed of environment variables, matching the following regular +expression:: + + \$\{[A-Za-z0-9-_]\}:\$\{[A-Za-z0-9-_]\} + +When ``url`` refers to a VCS repository: + +- A ``vcs`` field MUST be present, containing the name of the VCS + (i.e. one of git, hg, bzr, svn). Other VCS SHOULD be registered by + amending this PEP. +- The ``url`` value MUST be compatible with the corresponding VCS, + so an installer can hand it off without transformation to a + checkout/download command of the VCS. +- If the VCS supports commit-hash based revision identifiers, a ``commit-hash`` + field MUST be set by the installer in order to reference the immutable + version of the source code that was installed. +- In addition, a ``ref`` field MAY be present to reference a + branch/tag/revision compatible with the VCS. + +When ``url`` is a direct reference to a source archive or wheel: + +- A ``hash`` field SHOULD be present, with value + ``=``. + It is RECOMMENDED that only hashes which are unconditionally provided by + the latest version of the standard library's ``hashlib`` module be used for + source archive hashes. At time of writing, that list consists of 'md5', + 'sha1', 'sha224', 'sha256', 'sha384', and 'sha512'. + +A ``subdirectory`` field MAY be present containing a directory path, +relative to the root of the VCS repository or source archive, +to specify where ``pyproject.toml`` or ``setup.py`` is located. + +Examples +======== + +Example source_url.json +----------------------- + +Source archive: + +.. code:: + + { + "url": "https://github.com/pypa/pip/archive/1.3.1.zip", + "hash": "sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8" + } + +Git URL with tag and commit hash: + +.. code:: + + { + "url": "https://github.com/pypa/pip.git", + "vcs": "git", + "ref": "1.3.1", + "commit-hash": "7921be1537eac1e97bc40179a57f0349c2aee67d" + } + +Example pip commands and their effect on source_url.json +-------------------------------------------------------- + +Commands that generate a ``source_url.json``: + +* pip install https://example.com/app-1.0.tgz +* pip install https://example.com/app-1.0.whl +* pip install "git+https://example.com/repo/app.git#egg=app&subdirectory=setup" +* pip install ./app +* pip instal file:///home/user/app + +Commands that *do not* generate a ``source_url.json`` + +* pip install app +* pip install app --no-index --find-links https://example.com/ +* pip install --editable "git+https://example.com/repo/app.git#egg=app&subdirectory=setup" +* pip install -e ./app + +Use cases +========= + +"Freezing" an environment + + Tools, such as ``pip freeze``, which generate requirements from the Database + of Installed Python Distributions SHOULD exploit ``source_url.json`` + if it is present, and give it priority over the Version metadata in order + to generate a higher fidelity output. In presence of a ``vcs`` source URL, + The ``commit_hash`` field SHOULD be used in priority in order to provided + the highest possible fidelity to the originally installed version. If + supported by it's requirement format (such as `PEP440 Direct References`_), + tools are encouraged to output both ``ref``and ``commit_hash``. + Tools MAY choose another approach, depending on the needs of their users. + +Backwards Compatibility +======================= + +Since this PEP specifies a new file in the ``.dist-info`` directory, +there are no backwards compatibility implications. + +Alternatives +============ + +PEP426 source_url +----------------- + +The now withdrawn PEP 426 specifies a ``source_url`` metadata entry. +It is also implemented in `distlib`_. + +This format lacks support for the ``subdirectory`` option of pip requirement +URLs. The same limitation is present in PEP440 direct references. + +It also lacks explicit support for `environment variables in the user:password +part of URLs`_. + +The introduction of a key/value extensibility mechanism and support +for environment variables for user:password in PEP440, would allow to use that +specification for ``source_url``. + +References +========== + +.. _`pip issue #609`: https://github.com/pypa/pip/issues/609 +.. _`thread on discuss.python.org`: https://discuss.python.org/t/pip-freeze-vcs-urls-and-pep-517-feat-editable-installs/1473 +.. _PEP440: http://www.python.org/dev/peps/pep-0440 +.. _`VCS URLs supported by pip`: https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support +.. _`PEP440 Direct References`: https://www.python.org/dev/peps/pep-0440/#direct-references +.. _`Pipfile and Pipfile.lock`: https://github.com/pypa/pipfile +.. _distlib: https://distlib.readthedocs.io +.. _`environment variables in the user:password part of URLs`: https://pip.pypa.io/en/stable/reference/pip_install/#id10 + +Copyright +========= + +This document has been placed in the public domain. + + + .. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From b4f2233b6e64a8252d80c89ece545f007edc5a92 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul=20=28ACSONE=29?= Date: Wed, 24 Apr 2019 19:32:42 +0200 Subject: [PATCH 2/7] source_url -> direct_url as suggested by ncoghlan plus: - clarify the pep title - add a word about the original intent of PEP 426 source url - some spelling and grammar --- pep-9999.rst | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 6d76067e5e4..7820de37cb0 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -1,5 +1,5 @@ PEP: 9999 -Title: Recording the source URL of an installed distribution +Title: Recording the origin of distributions installed from direct URL references Author: Stéphane Bidoul Sponsor: ??? Status: Draft @@ -12,7 +12,7 @@ Discussion-To: https://discuss.python.org/t/recording-the-source-url-of-an-insta Abstract ======== -This PEP specifies a ``source_url.json`` file in the Database of Installed +This PEP specifies a ``direct_url.json`` file in the Database of Installed Python Distributions PEP 376. The purpose of this file is to record, in an extensible manner, the origin of distributions that were installed from direct URL references (as opposed to distributions that were installed @@ -124,7 +124,7 @@ unrelated to editable installs. Rationale ========= -This PEP specifies a new ``source_url.json`` metadata file in the .dist-info +This PEP specifies a new ``direct_url.json`` metadata file in the .dist-info directory of an installed distribution. The fields specified are sufficient to reproduce the source archive and `VCS @@ -132,7 +132,7 @@ URLs supported by pip`_. They are also sufficient to reproduce `PEP440 Direct References`_, as well as `Pipfile and Pipfile.lock`_ entries. Since at least the above 3 different way to encode the information exist, -this PEP uses a key-value format, to not make any assumption on how a source +this PEP uses a key-value format, to not make any assumption on how a direct URL must ultimately be encoded in a requirement or lockfile. See also the `Alternatives`_ section below for more discussion about this choice. @@ -145,7 +145,7 @@ The json format allows for the addition of additional fields in the future. Specification ============= -This PEP specifies a ``source_url.json`` file in the ``.dist-info`` directory +This PEP specifies a ``direct_url.json`` file in the ``.dist-info`` directory of an installed distribution. This file MUST be created by installers when installing a distribution @@ -194,7 +194,7 @@ to specify where ``pyproject.toml`` or ``setup.py`` is located. Examples ======== -Example source_url.json +Example direct_url.json ----------------------- Source archive: @@ -217,10 +217,10 @@ Git URL with tag and commit hash: "commit-hash": "7921be1537eac1e97bc40179a57f0349c2aee67d" } -Example pip commands and their effect on source_url.json +Example pip commands and their effect on direct_url.json -------------------------------------------------------- -Commands that generate a ``source_url.json``: +Commands that generate a ``direct_url.json``: * pip install https://example.com/app-1.0.tgz * pip install https://example.com/app-1.0.whl @@ -228,7 +228,7 @@ Commands that generate a ``source_url.json``: * pip install ./app * pip instal file:///home/user/app -Commands that *do not* generate a ``source_url.json`` +Commands that *do not* generate a ``direct_url.json`` * pip install app * pip install app --no-index --find-links https://example.com/ @@ -241,12 +241,12 @@ Use cases "Freezing" an environment Tools, such as ``pip freeze``, which generate requirements from the Database - of Installed Python Distributions SHOULD exploit ``source_url.json`` + of Installed Python Distributions SHOULD exploit ``direct_url.json`` if it is present, and give it priority over the Version metadata in order - to generate a higher fidelity output. In presence of a ``vcs`` source URL, - The ``commit_hash`` field SHOULD be used in priority in order to provided + to generate a higher fidelity output. In presence of a ``vcs`` direct URL, + The ``commit_hash`` field SHOULD be used in priority in order to provide the highest possible fidelity to the originally installed version. If - supported by it's requirement format (such as `PEP440 Direct References`_), + supported by their requirement format (such as `PEP440 Direct References`_), tools are encouraged to output both ``ref``and ``commit_hash``. Tools MAY choose another approach, depending on the needs of their users. @@ -265,6 +265,8 @@ PEP426 source_url The now withdrawn PEP 426 specifies a ``source_url`` metadata entry. It is also implemented in `distlib`_. +It was intended for a slightly different purpose, for use in sdists. + This format lacks support for the ``subdirectory`` option of pip requirement URLs. The same limitation is present in PEP440 direct references. @@ -272,8 +274,8 @@ It also lacks explicit support for `environment variables in the user:password part of URLs`_. The introduction of a key/value extensibility mechanism and support -for environment variables for user:password in PEP440, would allow to use that -specification for ``source_url``. +for environment variables for user:password in PEP440, would be necessary +for use in this PEP. References ========== From c319130e147ff20dd83a7065a4db871ddefe089e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul=20=28ACSONE=29?= Date: Mon, 29 Apr 2019 17:47:01 +0200 Subject: [PATCH 3/7] improve abstract with language from pf_moore --- pep-9999.rst | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 7820de37cb0..38fcbc08afc 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -12,26 +12,31 @@ Discussion-To: https://discuss.python.org/t/recording-the-source-url-of-an-insta Abstract ======== -This PEP specifies a ``direct_url.json`` file in the Database of Installed -Python Distributions PEP 376. The purpose of this file is to record, in an -extensible manner, the origin of distributions that were installed -from direct URL references (as opposed to distributions that were installed -from a package index). +Following PEP 440, a distribution can be identified by a name and either a +version, or a direct reference (see `PEP440 Direct References`_). +After installation, the name and version are captured in the project metadata, +but currently there is no way to obtain details of the URL used when the +distribution was identified by a direct reference. -The main motivation of this PEP is allowing tools attempting to "freeze" the -state of a python environment to work in a broader range of situations. +This proposal defines +additional metadata, to be added to the installed distribution by the +installation front end, which records the direct reference for use by +consumers which introspect the database of installed packages (see PEP 376). Motivation ========== +The main motivation of this PEP is allowing tools attempting to "freeze" the +state of a python environment to work in a broader range of situations. + This PEP originated from the need to implement `pip issue #609`_: i.e. improving the behavior of ``pip freeze`` in presence of distributions installed from direct URL references. It follows a `thread on discuss.python.org`_ about the best course of action to implement it. -Installation from URL ---------------------- +Installation from direct references +----------------------------------- Python installers such as pip are capable of downloading and installing distributions from package indexes. They are also capable of downloading From 52193664c0d417f9ee16e1c8646d7cfa7d2cc056 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul=20=28ACSONE=29?= Date: Sun, 30 Jun 2019 11:49:00 +0200 Subject: [PATCH 4/7] improvements following discussions with cjerdonek --- pep-9999.rst | 200 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 172 insertions(+), 28 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 38fcbc08afc..6822a6a34f9 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -1,26 +1,26 @@ PEP: 9999 -Title: Recording the origin of distributions installed from direct URL references +Title: Recording the Direct URL Origin of installed distributions Author: Stéphane Bidoul -Sponsor: ??? +Sponsor: Chris Jerdonek +Discussions-To: https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535 Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 21-Apr-2019 Post-History: -Discussion-To: https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535 Abstract ======== Following PEP 440, a distribution can be identified by a name and either a -version, or a direct reference (see `PEP440 Direct References`_). +version, or a direct URL reference (see `PEP440 Direct References`_). After installation, the name and version are captured in the project metadata, but currently there is no way to obtain details of the URL used when the -distribution was identified by a direct reference. +distribution was identified by a direct URL reference. This proposal defines additional metadata, to be added to the installed distribution by the -installation front end, which records the direct reference for use by +installation front end, which records the Direct URL Origin for use by consumers which introspect the database of installed packages (see PEP 376). Motivation @@ -35,8 +35,8 @@ installed from direct URL references. It follows a `thread on discuss.python.org`_ about the best course of action to implement it. -Installation from direct references ------------------------------------ +Installation from direct URL references +--------------------------------------- Python installers such as pip are capable of downloading and installing distributions from package indexes. They are also capable of downloading @@ -118,9 +118,9 @@ loses trace of the origin after installation. Indeed editable installs implicitly record the VCS origin in the checkout directory, so the information can be recovered when running "freeze". -The use of this workaround; although useful, is fragile, creates confusion +The use of this workaround, although useful, is fragile, creates confusion about the purpose of the editable mode, and works only when the distribution -can be installed with setuptools (i.e. it is not usable with other pep517 +can be installed with setuptools (i.e. it is not usable with other PEP 517 build backends). For the sake of clarity, it is important to note that this PEP is otherwise @@ -138,7 +138,7 @@ URLs supported by pip`_. They are also sufficient to reproduce Since at least the above 3 different way to encode the information exist, this PEP uses a key-value format, to not make any assumption on how a direct -URL must ultimately be encoded in a requirement or lockfile. See also +URL reference must ultimately be encoded in a requirement or lockfile. See also the `Alternatives`_ section below for more discussion about this choice. Information has been taken from Ruby's bundler manual to verify it has similar @@ -151,7 +151,7 @@ Specification ============= This PEP specifies a ``direct_url.json`` file in the ``.dist-info`` directory -of an installed distribution. +of an installed distribution, to record the Direct URL Origin of the distribution. This file MUST be created by installers when installing a distribution from a requirement specifying a direct URL reference (including a VCS URL @@ -160,6 +160,10 @@ in *non*-editable mode). This file MUST NOT be created when installing a distribution from an other type of requirement (i.e. name plus version specifier, or URL in editable mode). +This json MUST be a flat dictionary where all keys and values are of string type. +For the sake of forward compatibility, tools SHOULD ignore values which are +not of string type. + If present, it MUST contain at least one field with name ``url``. ``url`` MUST be stripped of any sensitive authentication information, @@ -167,23 +171,27 @@ for security reasons. The user:password section of the URL MAY however be composed of environment variables, matching the following regular expression:: - \$\{[A-Za-z0-9-_]\}:\$\{[A-Za-z0-9-_]\} + \$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})? When ``url`` refers to a VCS repository: - A ``vcs`` field MUST be present, containing the name of the VCS - (i.e. one of git, hg, bzr, svn). Other VCS SHOULD be registered by + (i.e. one of ``git``, ``hg``, ``bzr``, ``svn``).Other VCS SHOULD be registered by amending this PEP. - The ``url`` value MUST be compatible with the corresponding VCS, so an installer can hand it off without transformation to a checkout/download command of the VCS. -- If the VCS supports commit-hash based revision identifiers, a ``commit-hash`` - field MUST be set by the installer in order to reference the immutable +- A ``requested_revision`` field MAY be present to reference the + branch/tag/ref/commit/revision (in a format compatible with the VCS) that + was requested for installation. +- A ``commit_id`` field MUST be present, containing the + exact commit/revision number that was installed. + If the VCS supports commit-hash + based revision identifiers, such commit-hash MUST be used as + ``commit_id`` in order to reference the immutable version of the source code that was installed. -- In addition, a ``ref`` field MAY be present to reference a - branch/tag/revision compatible with the VCS. -When ``url`` is a direct reference to a source archive or wheel: +When ``url`` refers to a source archive, a wheel, or a local directory: - A ``hash`` field SHOULD be present, with value ``=``. @@ -192,10 +200,131 @@ When ``url`` is a direct reference to a source archive or wheel: source archive hashes. At time of writing, that list consists of 'md5', 'sha1', 'sha224', 'sha256', 'sha384', and 'sha512'. +.. note:: + + When the requested URL points to a local directory that happens to contain a + VCS checkout, installers MUST NOT attempt to infer any VCS information and + therefore MUST NOT output any vcs related information (such as ``vcs`` field) + in ``direct_url.json``. + A ``subdirectory`` field MAY be present containing a directory path, -relative to the root of the VCS repository or source archive, +relative to the root of the VCS repository, source archive or local directory, to specify where ``pyproject.toml`` or ``setup.py`` is located. +.. note:: + + As a general rule, installers should as much as possible preserve the + information that was provided in the requested URL when generating + ``direct_url.json``. For example user:password environment variables + should be preserved and ``requested_revision`` should reflect the revision that was + provided in the requested URL as faithfully as possible. This information is + however *enriched* with more precise data, such as ``commit_id``. + +Registered VCS +-------------- + +This section lists the registered VCS, along with precisions on how +to use the ``vcs``, ``requested_revision`` and ``commit_id`` fields. +Tools MAY support other VCS although it is RECOMMENDED to register +them by amending this PEP. The ``vcs`` field SHOULD be the command name +(lowercased). Additional fields that would be necessary to +support such VCS SHOULD be prefixed with the VCS command name. + +Git ++++ + +Home page + + https://git-scm.com/ + +vcs command + + git + +vcs field + + git + +requested_revision field + + A tag name, branch name, git ref, commit hash, shortened commit hash. + +commit_id field + + A commit hash (40 hexadecimal characters sha1). + +Mercurial ++++++++++ + +Home page + + https://www.mercurial-scm.org/ + +vcs command + + hg + +vcs field + + hg + +requested_revision field + + A tag name, branch name, git ref, changeset ID, shortened changeset ID. + +commit_id field + + A changeset ID (40 hexadecimal characters). + +Bazaar +++++++ + +Home page + + https://bazaar.canonical.com/ + +vcs command + + bzr + +vcs field + + bzr + +requested_revision field + + A tag name, branch name, revision id. + +commit_id field + + A revision id. + +Subversion +++++++++++ + +Home page + + https://subversion.apache.org/ + +vcs command + + svn + +vcs field + + svn + +requested_revision field + + ``requested_revision`` must be compatible with ``svn checkout`` ``--revision`` option. + In Subversion, branch or tag is part of ``url``. + +commit_id + + Since Subversion does not support globally unique identifiers, + this field is the Subversion revision number in the corresponding + repository. + Examples ======== @@ -211,15 +340,15 @@ Source archive: "hash": "sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8" } -Git URL with tag and commit hash: +Git URL with tag and commit-hash: .. code:: { "url": "https://github.com/pypa/pip.git", "vcs": "git", - "ref": "1.3.1", - "commit-hash": "7921be1537eac1e97bc40179a57f0349c2aee67d" + "requested_revision": "1.3.1", + "commit_id": "7921be1537eac1e97bc40179a57f0349c2aee67d" } Example pip commands and their effect on direct_url.json @@ -231,7 +360,7 @@ Commands that generate a ``direct_url.json``: * pip install https://example.com/app-1.0.whl * pip install "git+https://example.com/repo/app.git#egg=app&subdirectory=setup" * pip install ./app -* pip instal file:///home/user/app +* pip install file:///home/user/app Commands that *do not* generate a ``direct_url.json`` @@ -248,11 +377,11 @@ Use cases Tools, such as ``pip freeze``, which generate requirements from the Database of Installed Python Distributions SHOULD exploit ``direct_url.json`` if it is present, and give it priority over the Version metadata in order - to generate a higher fidelity output. In presence of a ``vcs`` direct URL, - The ``commit_hash`` field SHOULD be used in priority in order to provide + to generate a higher fidelity output. In presence of a ``vcs`` direct URL reference, + The ``commit_id`` field SHOULD be used in priority in order to provide the highest possible fidelity to the originally installed version. If supported by their requirement format (such as `PEP440 Direct References`_), - tools are encouraged to output both ``ref``and ``commit_hash``. + tools are encouraged to output both ``requested_revision`` and ``commit_id``. Tools MAY choose another approach, depending on the needs of their users. Backwards Compatibility @@ -273,7 +402,7 @@ It is also implemented in `distlib`_. It was intended for a slightly different purpose, for use in sdists. This format lacks support for the ``subdirectory`` option of pip requirement -URLs. The same limitation is present in PEP440 direct references. +URLs. The same limitation is present in `PEP440 Direct References`_. It also lacks explicit support for `environment variables in the user:password part of URLs`_. @@ -282,6 +411,13 @@ The introduction of a key/value extensibility mechanism and support for environment variables for user:password in PEP440, would be necessary for use in this PEP. +revision vs ref +--------------- + +The ``requested_revision`` key was retained over ``requested_ref`` as it is a more generic term +across various VCS and ``ref`` has a specific meaning for ``git``. + + References ========== @@ -294,6 +430,14 @@ References .. _distlib: https://distlib.readthedocs.io .. _`environment variables in the user:password part of URLs`: https://pip.pypa.io/en/stable/reference/pip_install/#id10 +Acknowledgements +================ + +Various people helped make this PEP a reality. Paul F. Moore provided the +essence of the abstract. Nick Coghlan suggested the direct_url name. Chris +Jerdonek made many insightful suggestions that helped clarifying several +aspects. + Copyright ========= From 07bd0df4f7248968c6bc3316c8600bb1a159389d Mon Sep 17 00:00:00 2001 From: Chris Jerdonek Date: Fri, 2 Aug 2019 10:32:07 +0200 Subject: [PATCH 5/7] add tag/branch/git_ref and their motivation Plus various language improvements. --- pep-9999.rst | 124 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 90 insertions(+), 34 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 6822a6a34f9..fe662757bd8 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -7,7 +7,7 @@ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 21-Apr-2019 -Post-History: +Post-History: Abstract ======== @@ -26,11 +26,12 @@ consumers which introspect the database of installed packages (see PEP 376). Motivation ========== -The main motivation of this PEP is allowing tools attempting to "freeze" the -state of a python environment to work in a broader range of situations. +The original motivation of this PEP was to permit tools with a "freeze" +operation allowing a Python environment to be recreated to work in a broader +range of situations. -This PEP originated from the need to implement `pip issue #609`_: -i.e. improving the behavior of ``pip freeze`` in presence of distributions +Specifically, the PEP originated from the desire to address `pip issue #609`_: +i.e. improving the behavior of ``pip freeze`` in the presence of distributions installed from direct URL references. It follows a `thread on discuss.python.org`_ about the best course of action to implement it. @@ -49,7 +50,7 @@ In other words two relevant installation modes exist. 1. the package to install is specified as a name and version specifier: In this case, the installer looks in a package index (or optionally - using --find-links in the case of pip) to find the distribution to install. + using ``--find-links`` in the case of pip) to find the distribution to install. 2. The package to install is specified as a direct URL reference: @@ -108,13 +109,37 @@ In such cases, it is important for "freeze" to pin the exact VCS reference (commit-hash if available) that was installed, in order to create reproducible builds with the highest possible fidelity. +Additional origin metadata available for VCS URLs +------------------------------------------------- + +For VCS URLs, there is additional origin information available only at +install time useful for introspection and certain workflows. For example, +when installing a revision from a VCS URL, a tool can determine if the +revision corresponds to a branch, tag or (in the case of Git) a ref. This +information can be used when introspecting the database of installed packages +to communicate to users more information about what version was installed +(e.g. whether a branch or tag was installed and, if so, the name of the +branch or tag). This also permits one to know whether a PEP 440 direct +reference URL can be constructed using the tag form, as only tags have the +semantics of immutability. + +In cases where the revision is mutable (e.g. branches and Git refs), knowing +this information enables workflows where users can e.g. update to the latest +version of a branch they are tracking, or update to the latest version of a +pull request they are reviewing locally. In contrast, when the revision is a +tag, tools can know in advance (e.g. without network calls) that no update is +needed. + +As with the URL itself, if this information isn't recorded at install time +when the VCS repository is available, it would otherwise be lost. + Note about "editable" installs ------------------------------ The editable installation mode of pip roughly lets a user insert a local directory in sys.path for development purpose. This mode is somewhat -abused to work around the fact that a non editable install from a VCS URL -loses trace of the origin after installation. +abused to work around the fact that a non-editable install from a VCS URL +loses track of the origin after installation. Indeed editable installs implicitly record the VCS origin in the checkout directory, so the information can be recovered when running "freeze". @@ -129,15 +154,19 @@ unrelated to editable installs. Rationale ========= -This PEP specifies a new ``direct_url.json`` metadata file in the .dist-info -directory of an installed distribution. +This PEP specifies a new ``direct_url.json`` metadata file in the +``.dist-info`` directory of an installed distribution. The fields specified are sufficient to reproduce the source archive and `VCS -URLs supported by pip`_. They are also sufficient to reproduce -`PEP440 Direct References`_, as well as `Pipfile and Pipfile.lock`_ entries. - -Since at least the above 3 different way to encode the information exist, -this PEP uses a key-value format, to not make any assumption on how a direct +URLs supported by pip`_. They are also sufficient to reproduce `PEP440 Direct +References`_, as well as `Pipfile and Pipfile.lock`_ entries. Finally, they +are sufficient to record the branch, tag, and/or Git ref origin of the +installed version that is already available for editable installs by virtue +of a VCS checkout being present. + +Since at least three different ways already exist to encode this type of +information, this PEP uses a key-value format, so as not to make any +assumption on how a direct URL reference must ultimately be encoded in a requirement or lockfile. See also the `Alternatives`_ section below for more discussion about this choice. @@ -145,7 +174,7 @@ Information has been taken from Ruby's bundler manual to verify it has similar capabilities and inform the selection and naming of fields in this specifications. -The json format allows for the addition of additional fields in the future. +The JSON format allows for the addition of additional fields in the future. Specification ============= @@ -154,13 +183,13 @@ This PEP specifies a ``direct_url.json`` file in the ``.dist-info`` directory of an installed distribution, to record the Direct URL Origin of the distribution. This file MUST be created by installers when installing a distribution -from a requirement specifying a direct URL reference (including a VCS URL -in *non*-editable mode). +from a requirement specifying a direct URL reference (including a VCS URL) +in *non*-editable mode. This file MUST NOT be created when installing a distribution from an other type of requirement (i.e. name plus version specifier, or URL in editable mode). -This json MUST be a flat dictionary where all keys and values are of string type. +This JSON MUST be a flat dictionary where all keys and values are of string type. For the sake of forward compatibility, tools SHOULD ignore values which are not of string type. @@ -176,20 +205,25 @@ expression:: When ``url`` refers to a VCS repository: - A ``vcs`` field MUST be present, containing the name of the VCS - (i.e. one of ``git``, ``hg``, ``bzr``, ``svn``).Other VCS SHOULD be registered by + (i.e. one of ``git``, ``hg``, ``bzr``, ``svn``). Other VCS's SHOULD be registered by amending this PEP. - The ``url`` value MUST be compatible with the corresponding VCS, so an installer can hand it off without transformation to a checkout/download command of the VCS. -- A ``requested_revision`` field MAY be present to reference the - branch/tag/ref/commit/revision (in a format compatible with the VCS) that - was requested for installation. +- A ``requested_revision`` field MAY be present naming a + branch/tag/ref/commit/revision/etc (in a format compatible with the VCS) + to install. - A ``commit_id`` field MUST be present, containing the exact commit/revision number that was installed. If the VCS supports commit-hash based revision identifiers, such commit-hash MUST be used as ``commit_id`` in order to reference the immutable version of the source code that was installed. +- A ``tag`` field naming a tag MAY be present to indicate that a particular + tag was installed. +- A ``branch`` field naming a branch MAY be present to indicate that a + particular branch was installed. If ``branch`` is present, ``tag`` MUST not + be present. When ``url`` refers to a source archive, a wheel, or a local directory: @@ -204,7 +238,7 @@ When ``url`` refers to a source archive, a wheel, or a local directory: When the requested URL points to a local directory that happens to contain a VCS checkout, installers MUST NOT attempt to infer any VCS information and - therefore MUST NOT output any vcs related information (such as ``vcs`` field) + therefore MUST NOT output any vcs related information (such as the ``vcs`` field) in ``direct_url.json``. A ``subdirectory`` field MAY be present containing a directory path, @@ -223,9 +257,10 @@ to specify where ``pyproject.toml`` or ``setup.py`` is located. Registered VCS -------------- -This section lists the registered VCS, along with precisions on how -to use the ``vcs``, ``requested_revision`` and ``commit_id`` fields. -Tools MAY support other VCS although it is RECOMMENDED to register +This section lists the registered VCS's; expanded, VCS-specific information +on how to use the ``vcs``, ``requested_revision``, and other fields; and in +some cases additional VCS-specific fields. +Tools MAY support other VCS's although it is RECOMMENDED to register them by amending this PEP. The ``vcs`` field SHOULD be the command name (lowercased). Additional fields that would be necessary to support such VCS SHOULD be prefixed with the VCS command name. @@ -247,12 +282,33 @@ vcs field requested_revision field - A tag name, branch name, git ref, commit hash, shortened commit hash. + A tag name, branch name, Git ref, commit hash, shortened commit hash, + or other commit-ish. commit_id field A commit hash (40 hexadecimal characters sha1). +branch field + + If no ``requested_revision`` is provided and the remote repository has + a default branch, this field can be used to record the default branch that + was installed. + +VCS-specific fields: + +- A ``git_ref`` field naming a Git ref (string beginning with ``refs/``) MAY + be present to indicate that a particular ref was installed (e.g. + ``refs/pull/123/head``). + +.. note:: + + Installers can use the ``git show-ref`` and ``git symbolic-ref`` commands + to determine if the ``requested_revision`` corresponds to a Git ref. + In turn, a ref beginning with ``refs/tags/`` corresponds to a tag, and + a ref beginning with ``refs/remotes/origin/`` after cloning corresponds + to a branch. + Mercurial +++++++++ @@ -270,7 +326,7 @@ vcs field requested_revision field - A tag name, branch name, git ref, changeset ID, shortened changeset ID. + A tag name, branch name, changeset ID, shortened changeset ID. commit_id field @@ -377,11 +433,11 @@ Use cases Tools, such as ``pip freeze``, which generate requirements from the Database of Installed Python Distributions SHOULD exploit ``direct_url.json`` if it is present, and give it priority over the Version metadata in order - to generate a higher fidelity output. In presence of a ``vcs`` direct URL reference, - The ``commit_id`` field SHOULD be used in priority in order to provide + to generate a higher fidelity output. In the presence of a ``vcs`` direct URL reference, + the ``commit_id`` field SHOULD be used in priority in order to provide the highest possible fidelity to the originally installed version. If - supported by their requirement format (such as `PEP440 Direct References`_), - tools are encouraged to output both ``requested_revision`` and ``commit_id``. + supported by their requirement format, tools are encouraged also to output + the ``tag`` value if present, as it has immutable semantics. Tools MAY choose another approach, depending on the needs of their users. Backwards Compatibility @@ -408,7 +464,7 @@ It also lacks explicit support for `environment variables in the user:password part of URLs`_. The introduction of a key/value extensibility mechanism and support -for environment variables for user:password in PEP440, would be necessary +for environment variables for user:password in PEP 440, would be necessary for use in this PEP. revision vs ref From 9b9eb808123b1504942e69961591d0fbdda963f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul=20=28ACSONE=29?= Date: Fri, 2 Aug 2019 10:57:52 +0200 Subject: [PATCH 6/7] add Chris Jerdonek as co-author --- pep-9999.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index fe662757bd8..3221f6c023a 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -1,7 +1,6 @@ PEP: 9999 Title: Recording the Direct URL Origin of installed distributions -Author: Stéphane Bidoul -Sponsor: Chris Jerdonek +Author: Stéphane Bidoul , Chris Jerdonek Discussions-To: https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535 Status: Draft Type: Standards Track @@ -490,9 +489,7 @@ Acknowledgements ================ Various people helped make this PEP a reality. Paul F. Moore provided the -essence of the abstract. Nick Coghlan suggested the direct_url name. Chris -Jerdonek made many insightful suggestions that helped clarifying several -aspects. +essence of the abstract. Nick Coghlan suggested the direct_url name. Copyright ========= From e5e382c62520613ef9294ed122e1d173e135c2c8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Bidoul=20=28ACSONE=29?= Date: Thu, 15 Aug 2019 20:08:30 +0200 Subject: [PATCH 7/7] Use latest PEP licensing requirements Co-Authored-By: Aliaksei Urbanski --- pep-9999.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 3221f6c023a..5cce2034b97 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -494,7 +494,8 @@ essence of the abstract. Nick Coghlan suggested the direct_url name. Copyright ========= -This document has been placed in the public domain. +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. ..