[Format][Docs] Clarify (remove?) usage of the term "logical types"

[jorisvandenbossche]

In several places in the Arrow specification and documentation we use the term "logical types", although we don't use it consistently and we don't actually have physical types (only physical layouts) to contrast it with.

Current usage

The Columnar Format doc page has a section called "Logical Types" (https://arrow.apache.org/docs/15.0/format/Columnar.html#logical-types) to contrast those types from the physical layouts:

The Schema.fbs defines built-in logical types supported by the Arrow columnar format. Each logical type uses one of the above physical layouts. Nested logical types may have different physical layouts depending on the particular realization of the type.

It explains an Array as having a logical data type, where "Each logical data type has a well-defined physical layout."

The authoritative Schema.fbs also uses the term:

arrow/format/Schema.fbs

Line 18 in 07a30d9

/// Logical types, vector layouts, and schemas

although it uses the term also in a "correct" way (but incorrect in the way we define the term currently):

arrow/format/Schema.fbs

Lines 101 to 105 in 07a30d9

	/// Represents the same logical types that List can, but contains offsets and
	/// sizes allowing for writes in any order and sharing of child values among
	/// list values.
	table ListView {
	}

The Python docs (https://arrow.apache.org/docs/15.0/python/data.html#type-metadata):

We use the name logical type because the physical storage may be the same for one or more types. For example, int64, float64, and timestamp[ms] all occupy 64 bits per value.

Further, in various implementations the term is obviously used as well.

In the Terminology section of the Columnar Format docs (https://arrow.apache.org/docs/15.0/format/Columnar.html#terminology), we define it as:

Logical type: An application-facing semantic value type that is implemented using some physical layout. For example, Decimal values are stored as 16 bytes in a fixed-size binary layout. Similarly, strings can be stored as List<1-byte>. A timestamp may be stored as 64-bit fixed-size layout.

which is mostly correct with our current usage ("using some physical layout"), but it is also confusing that it explains strings as List<1-byte> as we have a different physical layout used for strings

Previous discussion

Generally we use the term relatively consistently to contrast "logical types" from the "physical layouts", but confusion around the terminology has come up regularly (what are "physical types" then? And extension types are essentially "logical types", but annotating our own logical types). This was specifically discussed in #14752.

@amoeba proposed (#14752 (comment)):

Still some discussion to be had about avoiding "logical" vs. "physical" in favor of "types" and "layouts" and possibly updating the format docs comprehensively

[paleolimbot]

I was definitely confused by the term "logical type" when I became involved with Arrow several years ago and came slowly to the understanding that every system seems to use the terms logical and physical in a slightly different way and that it is more of a spectrum than a dichotomy.

We also have the term "encoding" used in two different ways: "run end encoded" is a "type" but "dictionary" is basically a special case in Schema.fbs and in the Arrow C Data interface. Implementations typically present these both as "types" (e.g., pa.dictionary()).

I agree that it is confusing but I am not sure what it should be replaced with. I suppose they could just be called "types" and "layouts", perhaps with dictionary encoding being a layout rather than a type?

[jorisvandenbossche]

I would maybe generally talk about "data types" to not just have "types". For the layouts I think the adjective "physical" is still useful, or otherwise more consistently use "memory" as adjective.
So that would give "physical layouts" or "memory layouts" and "data types"

[pitrou]

Issue resolved by pull request 41958
#41958