[C++][Docs] Document our ABI stability policy

[jorisvandenbossche]

With the 16.1.0 release, there have been a few issues related to whether this minor / feature release is expected to be ABI stable or expected to have bumped the so version.:

• [Packaging] soversion bumps on minor releases #41659
• [Packaging][deb] libarrow1600 shoud be libarrow1601 for 16.1.0 #41679
• Fix run exports to restrict the minor version conda-forge/arrow-cpp-feedstock#1409

Our release scripts do bump the so version for minor releases, as has been discussed and implemented around the time of the first 1.0.0 release (#4801, mailing list thread).

However, we don't document this policy anywhere AFAIK (and there was clearly also confusion about this among the committers as well). We should document this in the C++ docs somewhere, that the so version is bumped for every major and minor version, and thus consequently that bug-fix releases are expected to be ABI stable.

[vyasr]

This will be review for most people discussing here, but just wanted to summarize explicitly to make sure we're clear and all on the same page:

Status quo

• SOVERSION=${MAJOR}${MINOR} (so version 16.1.0 produces libarrow.so.1601, while version 16.0.1 produces libarrow.so.1600)
• Patch versions are promised to be ABI-compatible (but this is not explicitly verified in any way)
• Minor versions are not promised to be ABI-compatible
• Packaging metadata (for conda, haven't looked at others) claims that the packages are compatible between minor versions, so the runtime library is allowed to float across minor versions within the same major family.

Conservative solution

• SOVERSION=${MAJOR}${MINOR}${PATCH} (so version 16.1.0 produces libarrow.so.160100, while version 16.0.1 produces libarrow.so.160001)
• Assumes a completely unstable ABI
• Might be the most strictly accurate claim at the moment
• Would be quite painful for consumers

Compromise solution

• Same as the status quo, but packaging is updated to reflect the real ABI guarantees rather than the current misaligned state where packages promise wider ABI-compatibility than the compiled libraries in the package.

Desired state

• SOVERSION=${MAJOR} (so version 16.1.0 and version 16.0.1 produce libarrow.so.16)
• Both minor and patch versions are promised to be ABI-compatible
• This may not be possible for arrow right now.

Summary

If minor or patch releases can break ABI, then the SOVERSION must reflect that so that libraries built against libarrow have the appropriate requirements baked into the executable (e.g. DT_NEEDED entries in ELF files). Downstream of the library, packaging metadata (in particular, conda metadata as is being discussed in conda-forge/arrow-cpp-feedstock#1409) must account for this by setting runtime dependencies of dependents of libarrow to account for the right version of libarrow at runtime. Making a decision on #41707 (i.e. clarifying the intended stability promises) is the first thing to be done, then we can work on updating packaging files.