PEP 639: Add custom IDs issue & clarify rejected license key str value

Author: CAM-Gerlach

pep-0639.rst

@@ -6,7 +6,7 @@ Author: Philippe Ombredanne <pombredanne at nexb.com>,
         C.A.M. Gerlach <CAM.Gerlach at Gerlach.CAM>
 Sponsor: Paul Moore <p.f.moore at gmail.com>
 PEP-Delegate: Paul Moore <p.f.moore at gmail.com>
-Discussions-To: https://discuss.python.org/t/2154
+Discussions-To: https://discuss.python.org/t/12622
 Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
@@ -1398,25 +1398,58 @@ key, retaining the ``expression`` subkey in the ``license`` table, or
 allowing both. Indeed, this would seem to have been envisioned by PEP 621
 itself with this PEP in mind, in particular the first approach::
 
-    A practical string value for the license key has been purposefully left out
-    to allow for a future PEP to specify support for SPDX expressions.
+    A practical string value for the license key has been purposefully left
+    out to allow for a future PEP to specify support for SPDX expressions
+    (the same logic applies to any sort of "type" field specifying what
+    license the file or text represents).
 
 However, while a working draft temporarily explored this solution, it was
 ultimately rejected, as it shared most of the downsides identified with
 adding new subkeys under the existing ``license`` table, as well as several
 of its own, with again minimal advantage over separating both.
 
-In particular, it means the top-level ``license`` key still maps to multiple
+Most importantly, it still means that per PEP 621, it is not possible to
+separately mark the ``[project]`` keys corresponding to the ``License`` and
+``License-Expression`` metadata fields as dynamic. This, in turn, still
+renders specifying metadata following that standard incompatible with
+conversion of legacy metadata, as specified in this PEP's
+`Converting legacy metadata`_ section, as PEP 621 strictly prohibits the
+``license`` key from being both present (to define the existing value of
+the ``License`` field, or the path to a license file, and thus able to be
+converted), and specified as ``dynamic`` (which would allow tools to
+use the generated value for the ``License-Expression`` field.
+
+For the same reasons, this would make it impossible to back-fill the
+``License`` field from the ``License-Expression`` field as this PEP
+currently allows (without making an exception from strict
+``dynamic`` behavior in this case), as again, marking ``license`` as dynamic
+would mean it cannot be specified in the ``project`` table at all.
+
+Furthermore, this would mean existing project source metadata specifying
+``license`` as ``dynamic`` would be ambiguous, as it would be impossible for
+tools to statically determine if they are intended to conform to previous
+metadata versions specifying ``License``, or this version specifying
+``License-Expression``. Tools would have no way of determining which field,
+if either, might be filled in the resulting distribution's core metadata.
+By contrast, the present approach makes clear what the author intended,
+allows tools to unambiguously determine which field(s) may be dynamically
+inserted, and ensures backward compatibility such that current project
+source metadata do not unknowingly specify both the old and the new field
+as dynamic, and instead must do so explicitly per PEP 621's intent.
+
+Additionally, while differences from existing tool formats (and core metadata
+field names) has precedent in PEP 621 (though is best avoided if practical),
+using a key with an identical name as in all current tools (and of an existing
+core metadata field) to mean something different (and map to a different
+core metadata field), with distinct and incompatible syntax and semantics,
+does not, and is likely to create substantial and confusion and ambiguity
+for readers and authors, contrary to the fundamental goals of this PEP.
+
+Finally, this means that the top-level ``license`` key still maps to multiple
 core metadata fields with different purposes and interpretation (``License``
-and ``License-Expression``), one deprecated and one new, and still prevents
-them from being separately marked as dynamic, and conflates the same with
-an existing mark. This further exhibits the same divergence from both
-PEP 621, core metadata, tool file formats and the consensus in the discussion
-in not making the new license expression map to a corresponding new field,
-none of which was the case at the time PEP 621 was drafted.
-Finally, this would deny a clear separation from the old behavior by not
-cleanly deprecating the entire ``license`` key, and increases the complexity
-of the specification and implementation.
+and ``License-Expression``), this would deny a clear separation from the
+old behavior by not cleanly deprecating the ``license`` key, and
+increases the complexity of the specification and implementation.
 
 In addition to the aforementioned issues, this also requires deciding between
 the three individual approaches (``expression`` subkey, top-level string or
@@ -1441,8 +1474,7 @@ while adding even more spec and tool complexity and making there more than
 
 Therefore, a separate top-level ``license-expression`` key was adopted to avoid
 all these issues, with relatively minimal downside aside from adding a single
-additional top-level key and (versus some approaches) a few extra characters
-to type.
+additional key and (versus some approaches) a few extra characters to type.
 
 
 Add a ``type`` key to treat as expression
@@ -1999,6 +2031,75 @@ be reversed once a breaking revision of the metadata spec is issued)?
 Or should this not be explicitly allowed, discouraged or even prohibited?
 
 
+Should custom license identifiers be allowed?
+---------------------------------------------
+
+The current version of this PEP retains the behavior of only specifying
+the use of SPDX-defined license identifiers, as well as the explicitly defined
+custom identifiers ``LicenseRef-Public-Domain`` and ``LicenseRef-Proprietary``
+to handle the two common cases where projects have a license, but it is not
+one that has a recognized SPDX license identifier.
+
+For maximum flexibility, custom ``LicenseRef-<CUSTOM-TEXT>`` license
+identifiers could be allowed, which could potentially be useful for niche
+cases or corporate environments where ``LicenseRef-Proprietary`` is not
+appropriate or insufficiently specific, but relying on mainstream Python
+build tooling and the ``License-Expression`` metadata field is still
+desirable to use for this purpose.
+
+This has the downsides, however, of not catching misspellings of the
+canonically defined license identifiers and thus producing license metadata
+that is not a valid match for what the author intended, as well as users
+potentially thinking they have to prepend ``LicenseRef`` in front of valid
+license identifiers, as there seems to be some previous confusion about.
+Furthermore, this encourages the proliferation of bespoke license identifiers,
+which obviates the purpose of enabling clear, unambiguous and well
+understood license metadata for which this PEP was created.
+
+Indeed, for niche cases that need specific, proprietary custom licenses,
+they could always simply specify ``LicenseRef-Proprietary``, and then
+include the actual license files needed to unambiguously identify the license
+regardless (if not using SPDX license identifiers) under the ``License-File``
+fields. Requiring standards-conforming tools to allow custom license
+identifiers does not seem very useful, since standard tools will not recognize
+bespoke ones or know how to treat them. By contrast, bespoke tools, which
+would be required in any case to understand and act on custom identifiers,
+are explicitly allowed, with good reason (thus the ``SHOULD`` keyword)
+to not require that license identifiers conform to those listed here.
+Therefore, this specification still allows such use in private corporate
+environments or specific ecosystems, while avoiding the disadvantages of
+imposing them on all mainstream packaging tools.
+
+As an alternative, a literal ``LicenseRef-Custom`` identifier could be
+defined, which would more explicitly indicate that the license cannot be
+expressed with defined identifiers and the license text should be referenced
+for details, without carrying the negative and potentially inappropriate
+implications of ``LicenseRef-Proprietary``. This would avoid the main
+mentioned downsides (misspellings, confusion, license proliferation) of
+the approve approach of allowing an arbitrary ``LicenseRef``, while
+addressing several of the potential theoretical scenarios cited for it.
+
+On the other hand, as SPDX aims to (and generally does) encompass all
+FSF-recognized "Free" and OSI-approved "Open Source" licenses,
+and those sources are kept closely in sync and are now relatively stable,
+anything outside those bounds would generally be covered by
+``LicenseRef-Proprietary``, thus making ``LicenseRef-Custom`` less specific
+in that regard, and somewhat redundant to it. Furthermore, it may mislead
+authors of projects with complex/multiple licenses that they should use it
+over specifying a license expression.
+
+At present, the PEP retains the existing approach over either of these, given
+the use cases and benefits were judged to be sufficiently marginal based
+on the current understanding of the packaging landscape. For both these
+proposals, however, if more concrete use cases emerge, this can certainly
+be reconsidered, either for this current PEP or a future one (before or
+in tandem with actually removing the legacy unstructured ``License``
+metadata field). Not defining this now enables allowing it later
+(or still now, with custom packaging tools), without affecting backward
+compatibility, while the same is not so if they are allowed now and later
+determined to be unnecessary or too problematic in practice.
+
+
 Appendix 1. License Expression Examples
 =======================================