From a0d35a879b51ee9251efc9e5248342473fce2704 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Mon, 17 Apr 2023 14:35:22 +0900 Subject: [PATCH 1/8] GH-35084: [Docs][Format] Add how to change format specification This is based on the discussion: [DISCUSS] Format changes: process and requirements https://lists.apache.org/thread/9t0pglrvxjhrt4r4xcsc1zmgmbtr8pxj --- docs/source/format/Changing.rst | 67 +++++++++++++++++++++++++++++++++ docs/source/index.rst | 1 + 2 files changed, 68 insertions(+) create mode 100644 docs/source/format/Changing.rst diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst new file mode 100644 index 00000000000..b2b8c1599a5 --- /dev/null +++ b/docs/source/format/Changing.rst @@ -0,0 +1,67 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +How to Change Format Specification +================================== + +**Cross-language** is important in Apache Arrow. To keep it, we use +the following process when dealing with format (files in +``_) changes: + +* We must go through a formal DISCUSS and VOTE process +* We must have at least 2 reference implementations and associated + integration tests + +DISCUSS and VOTE Process +------------------------ + +DISCUSS process is a process to discuss the format changes in +public. Anyone can join the discussion. The DISCUSS process is done by +starting a thread in dev@arrow.apache.org with ``[DISCUSS]`` prefixed +subject. + +.. note:: + + We sometimes use ``[Discuss]``, ``DISCUSS: `` or something but + ``[DISCUSS]`` is recommended. + +Here are examples: + +* `[Discuss][Format] Add 32-bit and 64-bit Decimals `_ +* `[DISCUSS][Format] Starting to do some concrete work on the new "StringView" columnar data type `_ + +VOTE process is a process to tell whether we have reached +consensus. We can start a vote for the format changes after we reach +consensus in the preceding DISCUSS process. + +See also: `Apache Voting Process `_ + +At Least 2 Reference Implementations +------------------------------------ + +We must have at least 2 reference implementations and associated +integration tests to confirm whether the format changes are +cross-compatible and consistent. + +We must choose complete implementations for them. Here are candidate +implementations: + +* The C++ implementation +* The Java implementation + +We can discuss and vote to add more implementations to the list. +We may use doc:`../status` for the discussion. diff --git a/docs/source/index.rst b/docs/source/index.rst index 8341b9f3543..1813c201d3f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -82,6 +82,7 @@ target environment.** format/CStreamInterface format/ADBC format/Other + format/Changing format/Glossary .. _toc.development: From 456b85f199dc1684b878d7b22ae36cfb315ee89e Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Tue, 18 Apr 2023 14:19:22 +0900 Subject: [PATCH 2/8] Apply suggestions from code review Co-authored-by: Will Jones Co-authored-by: Weston Pace --- docs/source/format/Changing.rst | 60 ++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index b2b8c1599a5..744c9327d06 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -15,53 +15,65 @@ .. specific language governing permissions and limitations .. under the License. -How to Change Format Specification -================================== +Changing the Apache Arrow Format Specification +============================================== -**Cross-language** is important in Apache Arrow. To keep it, we use -the following process when dealing with format (files in -``_) changes: +**Cross-language** compatibility is important in Apache Arrow. To keep it, we use +the following process when dealing with changes to the format (files in +``_): -* We must go through a formal DISCUSS and VOTE process -* We must have at least 2 reference implementations and associated +* We must discuss and vote on the changes on the public mailing list +* We must have at least two reference implementations and associated integration tests -DISCUSS and VOTE Process ------------------------- +These do not have to be done in order. In most cases, having at least one +draft reference implementation is helpful for design discussion. -DISCUSS process is a process to discuss the format changes in -public. Anyone can join the discussion. The DISCUSS process is done by -starting a thread in dev@arrow.apache.org with ``[DISCUSS]`` prefixed -subject. +Discussion and Voting Process +----------------------------- + +Changes to the format should be discussed on the public mailing list. +Anyone can join the discussion. The discussion should be started by a +thread in dev@arrow.apache.org with the ``[DISCUSS]`` prefixed subject. .. note:: - We sometimes use ``[Discuss]``, ``DISCUSS: `` or something but + We sometimes use ``[Discuss]``, ``DISCUSS: `` or something similar but ``[DISCUSS]`` is recommended. -Here are examples: +Here are some examples: * `[Discuss][Format] Add 32-bit and 64-bit Decimals `_ * `[DISCUSS][Format] Starting to do some concrete work on the new "StringView" columnar data type `_ -VOTE process is a process to tell whether we have reached -consensus. We can start a vote for the format changes after we reach -consensus in the preceding DISCUSS process. +The voting process is used to verify we have reached consensus. +We can start a vote for the format changes after we reach +consensus in the preceding DISCUSS mailing list thread. Similar +to discussion threads, voting thread must have the subject prefix +``[VOTE]``. See also: `Apache Voting Process `_ -At Least 2 Reference Implementations ------------------------------------- +At Least Two Reference Implementations +-------------------------------------- -We must have at least 2 reference implementations and associated +We must have at least two reference implementations and associated integration tests to confirm whether the format changes are -cross-compatible and consistent. +compatible across languages and consistent. -We must choose complete implementations for them. Here are candidate +Reference implementations must be within complete Arrow implementations. Here are candidate implementations: * The C++ implementation * The Java implementation +* The Rust (arrow-rs) implementation +* The Go implementation We can discuss and vote to add more implementations to the list. -We may use doc:`../status` for the discussion. +We may use doc:`../status` to determine which implementations are complete. + +Versioning +---------- + +The format version (which is separate from the library versions) must also +be incremented as new changed are made. See :doc:`Versioning`. From 1fcd7fa2e53976e9b4d7da6194bbdad317e267ef Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 19 Apr 2023 05:19:45 +0900 Subject: [PATCH 3/8] Apply suggestions from code review Co-authored-by: Will Jones --- docs/source/format/Changing.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index 744c9327d06..9d53f4ec196 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -18,7 +18,7 @@ Changing the Apache Arrow Format Specification ============================================== -**Cross-language** compatibility is important in Apache Arrow. To keep it, we use +**Cross-language** compatibility is important in Apache Arrow. To maintain it, we use the following process when dealing with changes to the format (files in ``_): @@ -61,7 +61,9 @@ We must have at least two reference implementations and associated integration tests to confirm whether the format changes are compatible across languages and consistent. -Reference implementations must be within complete Arrow implementations. Here are candidate +Reference implementations must be within complete Arrow implementations. +For example, the C++ library is acceptable but the Python library is not, +since it is a wrapper around the C++ library. Here are candidate implementations: * The C++ implementation From 8693dd245c8ac6003b3da39c8ca044f0638a2218 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 19 Apr 2023 05:22:11 +0900 Subject: [PATCH 4/8] Fold and remove trailing spaces --- docs/source/format/Changing.rst | 44 +++++++++++++++++---------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index 9d53f4ec196..659c12e40da 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -18,23 +18,25 @@ Changing the Apache Arrow Format Specification ============================================== -**Cross-language** compatibility is important in Apache Arrow. To maintain it, we use -the following process when dealing with changes to the format (files in +**Cross-language** compatibility is important in Apache Arrow. To +maintain it, we use the following process when dealing with changes to +the format (files in ``_): * We must discuss and vote on the changes on the public mailing list * We must have at least two reference implementations and associated integration tests -These do not have to be done in order. In most cases, having at least one -draft reference implementation is helpful for design discussion. +These do not have to be done in order. In most cases, having at least +one draft reference implementation is helpful for design discussion. Discussion and Voting Process ----------------------------- Changes to the format should be discussed on the public mailing list. -Anyone can join the discussion. The discussion should be started by a -thread in dev@arrow.apache.org with the ``[DISCUSS]`` prefixed subject. +Anyone can join the discussion. The discussion should be started by a +thread in dev@arrow.apache.org with the ``[DISCUSS]`` prefixed +subject. .. note:: @@ -46,11 +48,10 @@ Here are some examples: * `[Discuss][Format] Add 32-bit and 64-bit Decimals `_ * `[DISCUSS][Format] Starting to do some concrete work on the new "StringView" columnar data type `_ -The voting process is used to verify we have reached consensus. -We can start a vote for the format changes after we reach -consensus in the preceding DISCUSS mailing list thread. Similar -to discussion threads, voting thread must have the subject prefix -``[VOTE]``. +The voting process is used to verify we have reached consensus. We can +start a vote for the format changes after we reach consensus in the +preceding DISCUSS mailing list thread. Similar to discussion threads, +voting thread must have the subject prefix ``[VOTE]``. See also: `Apache Voting Process `_ @@ -58,24 +59,25 @@ At Least Two Reference Implementations -------------------------------------- We must have at least two reference implementations and associated -integration tests to confirm whether the format changes are -compatible across languages and consistent. +integration tests to confirm whether the format changes are compatible +across languages and consistent. -Reference implementations must be within complete Arrow implementations. -For example, the C++ library is acceptable but the Python library is not, -since it is a wrapper around the C++ library. Here are candidate -implementations: +Reference implementations must be within complete Arrow +implementations. For example, the C++ library is acceptable but the +Python library is not, since it is a wrapper around the C++ +library. Here are candidate implementations: * The C++ implementation * The Java implementation * The Rust (arrow-rs) implementation * The Go implementation -We can discuss and vote to add more implementations to the list. -We may use doc:`../status` to determine which implementations are complete. +We can discuss and vote to add more implementations to the list. We +may use doc:`../status` to determine which implementations are +complete. Versioning ---------- -The format version (which is separate from the library versions) must also -be incremented as new changed are made. See :doc:`Versioning`. +The format version (which is separate from the library versions) must +also be incremented as new changed are made. See :doc:`Versioning`. From 1ad17718383beed5ab1def8fe0aab8cb9ff33f60 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 19 Apr 2023 14:15:57 +0900 Subject: [PATCH 5/8] Fix markup --- docs/source/format/Changing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index 659c12e40da..8caeb5c761c 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -40,7 +40,7 @@ subject. .. note:: - We sometimes use ``[Discuss]``, ``DISCUSS: `` or something similar but + We sometimes use ``[Discuss]``, ``DISCUSS:`` or something similar but ``[DISCUSS]`` is recommended. Here are some examples: From a21b22736fb89924dca3518077b9a9cd22260095 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 19 Apr 2023 14:16:44 +0900 Subject: [PATCH 6/8] Fix markup --- docs/source/format/Changing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index 8caeb5c761c..c42e1b26cb5 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -73,7 +73,7 @@ library. Here are candidate implementations: * The Go implementation We can discuss and vote to add more implementations to the list. We -may use doc:`../status` to determine which implementations are +may use :doc:`../status` to determine which implementations are complete. Versioning From 474a02314284cdffb1a990d38cfb50f0cfd98ee7 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 26 Apr 2023 09:12:25 +0900 Subject: [PATCH 7/8] Add a task about documentation explicitly --- docs/source/format/Changing.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index c42e1b26cb5..23e235d8f32 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -26,6 +26,9 @@ the format (files in * We must discuss and vote on the changes on the public mailing list * We must have at least two reference implementations and associated integration tests +* We must update the corresponding documentation (files in + ``_) + integration tests These do not have to be done in order. In most cases, having at least one draft reference implementation is helpful for design discussion. From dbd2c20a6206fe9209255627881bde5dec4ac43a Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Wed, 26 Apr 2023 09:14:36 +0900 Subject: [PATCH 8/8] Use note --- docs/source/format/Changing.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/source/format/Changing.rst b/docs/source/format/Changing.rst index 23e235d8f32..cd61af3b665 100644 --- a/docs/source/format/Changing.rst +++ b/docs/source/format/Changing.rst @@ -26,13 +26,16 @@ the format (files in * We must discuss and vote on the changes on the public mailing list * We must have at least two reference implementations and associated integration tests -* We must update the corresponding documentation (files in - ``_) - integration tests These do not have to be done in order. In most cases, having at least one draft reference implementation is helpful for design discussion. +.. note:: + + We must update the corresponding documentation (files in + ``_) + too. + Discussion and Voting Process -----------------------------