From 097f1ab64ce380a189bab88c16c7853f8b0f22a5 Mon Sep 17 00:00:00 2001 From: Xianjin Date: Thu, 11 Jan 2024 17:22:57 +0800 Subject: [PATCH 1/6] spec: multi-arg transform --- format/spec.md | 72 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 55 insertions(+), 17 deletions(-) diff --git a/format/spec.md b/format/spec.md index 80cdd6d2987f..319353eceb79 100644 --- a/format/spec.md +++ b/format/spec.md @@ -296,9 +296,9 @@ Data files are stored in manifests with a tuple of partition values that are use Tables are configured with a **partition spec** that defines how to produce a tuple of partition values from a record. A partition spec has a list of fields that consist of: -* A **source column id** from the table’s schema +* A **source column id** or a list of **source column ids** from the table’s schema * A **partition field id** that is used to identify a partition field and is unique within a partition spec. In v2 table metadata, it is unique across all partition specs. -* A **transform** that is applied to the source column to produce a partition value +* A **transform** that is applied to the source column or columns to produce a partition value * A **partition name** The source column, selected by id, must be a primitive type and cannot be contained in a map or list, but may be nested in a struct. For details on how to serialize a partition spec to JSON, see Appendix C. @@ -314,7 +314,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | Transform name | Description | Source types | Result type | |-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------| | **`identity`** | Source value, unmodified | Any | Source type | -| **`bucket[N]`** | Hash of value, mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | +| **`bucket[N]`** | Hash of value(s), mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | | **`truncate[W]`** | Value truncated to width `W` (see below) | `int`, `long`, `decimal`, `string` | Source type | | **`year`** | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | | **`month`** | Extract a date or timestamp month, as months from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | @@ -329,7 +329,7 @@ The `void` transform may be used to replace the transform in an existing partiti #### Bucket Transform Details -Bucket partition transforms use a 32-bit hash of the source value. The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0. +Bucket partition transforms use a 32-bit hash of the source value(s). The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0. Transforms are parameterized by a number of buckets [1], `N`. The hash mod `N` must produce a positive value by first discarding the sign bit of the hash value. In pseudo-code, the function is: @@ -337,11 +337,27 @@ Transforms are parameterized by a number of buckets [1], `N`. The hash mod `N` m def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N ``` +When bucket transform is applied on a list of values, the input is treated as concatenated bytes of each value. In pseudo-code, the function is: + +``` + def murmur3_x86_32_hashes(x1, x2, x3, ...) = { + byte[] bytes; + for (x in [x1, x2, x3, ...]) { + bytes = x == null ? bytes : append(bytes, bytesOf(x)); + } + return murmur3_x86_32_hash(bytes); + } + + def bucket_N(x1, x2, x3, ...) = (murmur3_x86_32_hashes(x1, x2, x3, ...) & Integer.MAX_VALUE) % N +``` + Notes: 1. Changing the number of buckets as a table grows is possible by evolving the partition spec. +2. `murmur3_x86_32_hashes` produces the same result as `murmur3_x86_32_hash` when applied on a single value. +3. NULL input in the list of values is ignored when computing the hash. If all the input values are NULL, NULL should be produced. -For hash function details by type, see Appendix B. +For hash function details by type and bytes representation of each type, see Appendix B. #### Truncate Transform Details @@ -383,8 +399,8 @@ Users can sort their data within partitions by columns to gain performance. The A sort order is defined by a sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of: -* A **source column id** from the table's schema -* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms). +* A **source column id** or a list of **source column ids** from the table's schema +* A **transform** that is used to produce values to be sorted on from the source column(s). This is the same transform as described in [partition transforms](#partition-transforms). * A **sort direction**, that can only be either `asc` or `desc` * A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last` @@ -1060,6 +1076,14 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho | **`float`** | `hashLong(doubleToLongBits(double(v))` [4]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` | | **`double`** | `hashLong(doubleToLongBits(v))` [4]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` | +For multi-arg hash function, the hash value is the result of `hashBytes` on the concatenation of the bytes of the arguments. NULL input is ignored. + +| Struct examples | Hash Specification | Test value | +|------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------| +| **`struct`** | `hashBytes(concatenation(littleEndianBytes(long(v)), utf8Bytes(b)))` | `{a: 34, b: "iceberg"}` ￫ `875336289` | +| **`struct`** | `hashBytes(concatenation(littleEndianBytes(microsecsFromUnixEpoch(b)),utf8Bytes(c)))` | `{b: '2017-11-16T22:31:08', c: "iceberg"}` ￫ `-1797269680` | + + Notes: 1. Integer and long hash results must be identical for all integer values. This ensures that schema evolution does not change bucket partition values if integer types are promoted. @@ -1119,21 +1143,29 @@ Partition specs are serialized as a JSON object with the following fields: Each partition field in the fields list is stored as an object. See the table for more detail: -|Transform or Field|JSON representation|Example| -|--- |--- |--- | -|**`identity`**|`JSON string: "identity"`|`"identity"`| -|**`bucket[N]`**|`JSON string: "bucket[]"`|`"bucket[16]"`| -|**`truncate[W]`**|`JSON string: "truncate[]"`|`"truncate[20]"`| -|**`year`**|`JSON string: "year"`|`"year"`| -|**`month`**|`JSON string: "month"`|`"month"`| -|**`day`**|`JSON string: "day"`|`"day"`| -|**`hour`**|`JSON string: "hour"`|`"hour"`| -|**`Partition Field`**|`JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}`|`{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}`| +| Transform or Field | JSON representation | Example | +|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **`identity`** | `JSON string: "identity"` | `"identity"` | +| **`bucket[N]`** | `JSON string: "bucket[]"` | `"bucket[16]"` | +| **`bucket[N]`** (multi-arg bucket [1]) | `JSON string: "bucketV2[]"` | `"bucketV2[16]"` | +| **`truncate[W]`** | `JSON string: "truncate[]"` | `"truncate[20]"` | +| **`year`** | `JSON string: "year"` | `"year"` | +| **`month`** | `JSON string: "month"` | `"month"` | +| **`day`** | `JSON string: "day"` | `"day"` | +| **`hour`** | `JSON string: "hour"` | `"hour"` | +| **`Partition Field`** | `JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}` | +| **`Partition Field with multi-arg transform`** [2] | `JSON object: {`
  `"source-id": -1,`
  `"source-ids": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": -1,`
  `"source-ids": [1,2],`
  `"field-id": 1000,`
  `"name": "id_type_bucket",`
  `"transform": "bucketV2[16]"`
`}` | In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec. The `field-id` property was added for each partition field in v2. In v1, the reference implementation assigned field ids sequentially in each spec starting at 1,000. See Partition Evolution for more details. +Notes: + +1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identifier this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. + This makes adding multi-arg transform a forward-compatible change, but not a backward-compatible change. +2. For partition fields with multi-arg transform, `source-id` is replaced by `source-ids` and marked as `-1` to be consistent with single-arg transform. `source-id` should still be emitted for single-arg transform. + ### Sort Orders Sort orders are serialized as a list of JSON object, each of which contains the following fields: @@ -1149,6 +1181,12 @@ Each sort field in the fields list is stored as an object with the following pro |--- |--- |--- | |**`Sort Field`**|`JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| +Similar with partition fields, sort fields could also contain multi source-ids for sorting: + +| Field | JSON representation | Example | +|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **`Sort Field(multi-arg transform)`** | `JSON object: {`
  `"transform": ,`
  `"source-id": -1,`
  `"source-ids": ,`
  `"direction": ,`
  `"null-order": `
`}` | `{`
  ` "transform": "bucketV2[4]",`
  ` "source-id": -1,`
  ` "source-id": [1,2],`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}` | + The following table describes the possible values for the some of the field within sort field: |Field|JSON representation|Possible values| From 6499552dee302f715dabdc55d15c22cb5fceb1e0 Mon Sep 17 00:00:00 2001 From: Xianjin Date: Thu, 11 Jan 2024 17:59:35 +0800 Subject: [PATCH 2/6] refine wording --- format/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 319353eceb79..d78d0cd440c9 100644 --- a/format/spec.md +++ b/format/spec.md @@ -298,7 +298,7 @@ Tables are configured with a **partition spec** that defines how to produce a tu * A **source column id** or a list of **source column ids** from the table’s schema * A **partition field id** that is used to identify a partition field and is unique within a partition spec. In v2 table metadata, it is unique across all partition specs. -* A **transform** that is applied to the source column or columns to produce a partition value +* A **transform** that is applied to the source column(s) to produce a partition value * A **partition name** The source column, selected by id, must be a primitive type and cannot be contained in a map or list, but may be nested in a struct. For details on how to serialize a partition spec to JSON, see Appendix C. @@ -1162,7 +1162,7 @@ The `field-id` property was added for each partition field in v2. In v1, the ref Notes: -1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identifier this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. +1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identify this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. This makes adding multi-arg transform a forward-compatible change, but not a backward-compatible change. 2. For partition fields with multi-arg transform, `source-id` is replaced by `source-ids` and marked as `-1` to be consistent with single-arg transform. `source-id` should still be emitted for single-arg transform. From 888c90bd22ddaefdbaad35553189b4099ca6c94d Mon Sep 17 00:00:00 2001 From: Xianjin YE Date: Sat, 13 Jan 2024 18:56:43 +0800 Subject: [PATCH 3/6] chore: address comments and refine wording Co-authored-by: Szehon Ho --- format/spec.md | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/format/spec.md b/format/spec.md index d78d0cd440c9..32f86428f1e1 100644 --- a/format/spec.md +++ b/format/spec.md @@ -337,15 +337,15 @@ Transforms are parameterized by a number of buckets [1], `N`. The hash mod `N` m def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N ``` -When bucket transform is applied on a list of values, the input is treated as concatenated bytes of each value. In pseudo-code, the function is: +When bucket transform is applied on a list of values, the hash is applied on the concatenated byte representations of all values. In pseudo-code, the function is: ``` def murmur3_x86_32_hashes(x1, x2, x3, ...) = { byte[] bytes; for (x in [x1, x2, x3, ...]) { - bytes = x == null ? bytes : append(bytes, bytesOf(x)); + if (x != null) bytes.append(bytesOf(x)) } - return murmur3_x86_32_hash(bytes); + return murmur3_x86_32_hash(bytes) } def bucket_N(x1, x2, x3, ...) = (murmur3_x86_32_hashes(x1, x2, x3, ...) & Integer.MAX_VALUE) % N @@ -1076,12 +1076,25 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho | **`float`** | `hashLong(doubleToLongBits(double(v))` [4]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` | | **`double`** | `hashLong(doubleToLongBits(v))` [4]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` | -For multi-arg hash function, the hash value is the result of `hashBytes` on the concatenation of the bytes of the arguments. NULL input is ignored. +For multiple arguments, hashBytes() is applied on the concatenated byte representation of each argument: -| Struct examples | Hash Specification | Test value | -|------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------| -| **`struct`** | `hashBytes(concatenation(littleEndianBytes(long(v)), utf8Bytes(b)))` | `{a: 34, b: "iceberg"}` ￫ `875336289` | -| **`struct`** | `hashBytes(concatenation(littleEndianBytes(microsecsFromUnixEpoch(b)),utf8Bytes(c)))` | `{b: '2017-11-16T22:31:08', c: "iceberg"}` ￫ `-1797269680` | +| Primitive type | Bytes representation | +|----------------------|------------------------------------------------| +| **`int`** | `littleEndianBytes(long(v))` | +| **`long`** | `littleEndianBytes(v)` | +| **`decimal(P,S)`** | `minBigEndian(unscaled(v))` | +| **`date`** | `littleEndianBytes(daysFromUnixEpoch(v))` | +| **`time`** | `littleEndianBytes(microsecsFromMidnight(v))` | +| **`timestamp`** | `littleEndianBytes(microsecsFromUnixEpoch(v))` | +| **`timestamptz`** | `littleEndianBytes(microsecsFromUnixEpoch(v))` | +| **`timestamp_ns`** | `littleEndianBytes(nanosecsFromUnixEpoch(v))` | +| **`timestamptz_ns`** | `littleEndianBytes(nanosecsFromUnixEpoch(v))` | +| **`string`** | `utf8Bytes(v)` | +| **`uuid`** | `uuidBytes(v)` | +| **`fixed(L)`** | `v` | +| **`binary`** | `v` | + +For example, the hash representation of `(a:int, b:string)` will be `hashBytes(concatenation(littleEndianBytes(long(v)), utf8Bytes(b))` Notes: @@ -1164,7 +1177,8 @@ Notes: 1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identify this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. This makes adding multi-arg transform a forward-compatible change, but not a backward-compatible change. -2. For partition fields with multi-arg transform, `source-id` is replaced by `source-ids` and marked as `-1` to be consistent with single-arg transform. `source-id` should still be emitted for single-arg transform. +2. For partition fields with a transform of multiple arguments, the ids of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. +3. For partition field with a transform with a single argument, the id of the source field is set on `source-id`, and `source-ids` is omitted. ### Sort Orders From a2f7b7aa010e6a628635261aaf0f990b328f94be Mon Sep 17 00:00:00 2001 From: Xianjin Date: Tue, 16 Jan 2024 19:35:27 +0800 Subject: [PATCH 4/6] address comments --- format/spec.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/format/spec.md b/format/spec.md index 32f86428f1e1..a3e12d482f87 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1166,8 +1166,8 @@ Each partition field in the fields list is stored as an object. See the table fo | **`month`** | `JSON string: "month"` | `"month"` | | **`day`** | `JSON string: "day"` | `"day"` | | **`hour`** | `JSON string: "hour"` | `"hour"` | -| **`Partition Field`** | `JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}` | -| **`Partition Field with multi-arg transform`** [2] | `JSON object: {`
  `"source-id": -1,`
  `"source-ids": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": -1,`
  `"source-ids": [1,2],`
  `"field-id": 1000,`
  `"name": "id_type_bucket",`
  `"transform": "bucketV2[16]"`
`}` | +| **`Partition Field`** [2] | `JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}` | +| **`Partition Field with multi-arg transform`** [3] | `JSON object: {`
  `"source-id": -1,`
  `"source-ids": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": -1,`
  `"source-ids": [1,2],`
  `"field-id": 1000,`
  `"name": "id_type_bucket",`
  `"transform": "bucketV2[16]"`
`}` | In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec. @@ -1177,8 +1177,8 @@ Notes: 1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identify this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. This makes adding multi-arg transform a forward-compatible change, but not a backward-compatible change. -2. For partition fields with a transform of multiple arguments, the ids of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. -3. For partition field with a transform with a single argument, the id of the source field is set on `source-id`, and `source-ids` is omitted. +2. For partition fields with a transform with a single argument, the id of the source field is set on `source-id`, and `source-ids` is omitted. +3. For partition fields with a transform of multiple arguments, the ids of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. ### Sort Orders @@ -1191,15 +1191,14 @@ Sort orders are serialized as a list of JSON object, each of which contains the Each sort field in the fields list is stored as an object with the following properties: -|Field|JSON representation|Example| -|--- |--- |--- | -|**`Sort Field`**|`JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| +| Field | JSON representation | Example | +|-----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **`Sort Field`** [1] | `JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}` | `{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}` | +| **`Sort Field with multi-arg transform`** [2] | `JSON object: {`
  `"transform": ,`
  `"source-id": -1,`
  `"source-ids": ,`
  `"direction": ,`
  `"null-order": `
`}` | `{`
  ` "transform": "bucketV2[4]",`
  ` "source-id": -1,`
  ` "source-id": [1,2],`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}` | -Similar with partition fields, sort fields could also contain multi source-ids for sorting: - -| Field | JSON representation | Example | -|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **`Sort Field(multi-arg transform)`** | `JSON object: {`
  `"transform": ,`
  `"source-id": -1,`
  `"source-ids": ,`
  `"direction": ,`
  `"null-order": `
`}` | `{`
  ` "transform": "bucketV2[4]",`
  ` "source-id": -1,`
  ` "source-id": [1,2],`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}` | +Notes: +1. For sort fields with a transform with a single argument, the id of the source field is set on `source-id`, and `source-ids` is omitted. +2. For sort fields with a transform of multiple arguments, the ids of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. The following table describes the possible values for the some of the field within sort field: From 0221fc906506eecd6b4bcb1e74ac0d33c2594f70 Mon Sep 17 00:00:00 2001 From: Xianjin Date: Thu, 18 Jan 2024 11:35:10 +0800 Subject: [PATCH 5/6] address comments --- format/spec.md | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/format/spec.md b/format/spec.md index a3e12d482f87..351c9072fe7b 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1076,7 +1076,7 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho | **`float`** | `hashLong(doubleToLongBits(double(v))` [4]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` | | **`double`** | `hashLong(doubleToLongBits(v))` [4]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` | -For multiple arguments, hashBytes() is applied on the concatenated byte representation of each argument: +For multiple arguments, `hashBytes()` is applied on the concatenated byte representation of each argument: | Primitive type | Bytes representation | |----------------------|------------------------------------------------| @@ -1094,7 +1094,7 @@ For multiple arguments, hashBytes() is applied on the concatenated byte represen | **`fixed(L)`** | `v` | | **`binary`** | `v` | -For example, the hash representation of `(a:int, b:string)` will be `hashBytes(concatenation(littleEndianBytes(long(v)), utf8Bytes(b))` +For example, the hash representation of `(a:int, b:string)` will be `hashBytes(concatenation(littleEndianBytes(long(v)), utf8Bytes(b))`. Notes: @@ -1156,18 +1156,18 @@ Partition specs are serialized as a JSON object with the following fields: Each partition field in the fields list is stored as an object. See the table for more detail: -| Transform or Field | JSON representation | Example | -|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **`identity`** | `JSON string: "identity"` | `"identity"` | -| **`bucket[N]`** | `JSON string: "bucket[]"` | `"bucket[16]"` | -| **`bucket[N]`** (multi-arg bucket [1]) | `JSON string: "bucketV2[]"` | `"bucketV2[16]"` | -| **`truncate[W]`** | `JSON string: "truncate[]"` | `"truncate[20]"` | -| **`year`** | `JSON string: "year"` | `"year"` | -| **`month`** | `JSON string: "month"` | `"month"` | -| **`day`** | `JSON string: "day"` | `"day"` | -| **`hour`** | `JSON string: "hour"` | `"hour"` | -| **`Partition Field`** [2] | `JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}` | -| **`Partition Field with multi-arg transform`** [3] | `JSON object: {`
  `"source-id": -1,`
  `"source-ids": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}` | `{`
  `"source-id": -1,`
  `"source-ids": [1,2],`
  `"field-id": 1000,`
  `"name": "id_type_bucket",`
  `"transform": "bucketV2[16]"`
`}` | +|Transform or Field|JSON representation|Example| +|--- |--- |--- | +|**`identity`**|`JSON string: "identity"`|`"identity"`| +|**`bucket[N]`**|`JSON string: "bucket[]"`|`"bucket[16]"`| +|**`bucket[N]`** (multi-arg bucket [1])| `JSON string: "bucketV2[]"` | `"bucketV2[16]"` | +|**`truncate[W]`**|`JSON string: "truncate[]"`|`"truncate[20]"`| +|**`year`**|`JSON string: "year"`|`"year"`| +|**`month`**|`JSON string: "month"`|`"month"`| +|**`day`**|`JSON string: "day"`|`"day"`| +|**`hour`**|`JSON string: "hour"`|`"hour"`| +|**`Partition Field`** [2]|`JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}`|`{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}`| +|**`Partition Field with multi-arg transform`** [3]|`JSON object: {`
  `"source-id": -1,`
  `"source-ids": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}`|`{`
  `"source-id": -1,`
  `"source-ids": [1,2],`
  `"field-id": 1000,`
  `"name": "id_type_bucket",`
  `"transform": "bucketV2[16]"`
`}`| In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec. @@ -1177,8 +1177,8 @@ Notes: 1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identify this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. This makes adding multi-arg transform a forward-compatible change, but not a backward-compatible change. -2. For partition fields with a transform with a single argument, the id of the source field is set on `source-id`, and `source-ids` is omitted. -3. For partition fields with a transform of multiple arguments, the ids of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. +2. For partition fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted. +3. For partition fields with a transform of multiple arguments, the IDs of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. ### Sort Orders @@ -1191,14 +1191,15 @@ Sort orders are serialized as a list of JSON object, each of which contains the Each sort field in the fields list is stored as an object with the following properties: -| Field | JSON representation | Example | -|-----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **`Sort Field`** [1] | `JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}` | `{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}` | -| **`Sort Field with multi-arg transform`** [2] | `JSON object: {`
  `"transform": ,`
  `"source-id": -1,`
  `"source-ids": ,`
  `"direction": ,`
  `"null-order": `
`}` | `{`
  ` "transform": "bucketV2[4]",`
  ` "source-id": -1,`
  ` "source-id": [1,2],`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}` | +|Field|JSON representation|Example| +|--- |--- |--- | +|**`Sort Field`** [1]|`JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| +|**`Sort Field with multi-arg transform`** [2]|`JSON object: {`
  `"transform": ,`
  `"source-id": -1,`
  `"source-ids": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucketV2[4]",`
  ` "source-id": -1,`
  ` "source-id": [1,2],`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| + Notes: -1. For sort fields with a transform with a single argument, the id of the source field is set on `source-id`, and `source-ids` is omitted. -2. For sort fields with a transform of multiple arguments, the ids of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. +1. For sort fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted. +2. For sort fields with a transform of multiple arguments, the IDs of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. The following table describes the possible values for the some of the field within sort field: From a8076dd10572a211dd4433338effaf095316c811 Mon Sep 17 00:00:00 2001 From: Xianjin Date: Wed, 24 Jan 2024 10:16:44 +0800 Subject: [PATCH 6/6] remove bucketV2 transform --- format/spec.md | 57 +++++++------------------------------------------- 1 file changed, 7 insertions(+), 50 deletions(-) diff --git a/format/spec.md b/format/spec.md index 351c9072fe7b..bc655c49dc57 100644 --- a/format/spec.md +++ b/format/spec.md @@ -314,7 +314,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | Transform name | Description | Source types | Result type | |-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------| | **`identity`** | Source value, unmodified | Any | Source type | -| **`bucket[N]`** | Hash of value(s), mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | +| **`bucket[N]`** | Hash of value, mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | | **`truncate[W]`** | Value truncated to width `W` (see below) | `int`, `long`, `decimal`, `string` | Source type | | **`year`** | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | | **`month`** | Extract a date or timestamp month, as months from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | @@ -329,7 +329,7 @@ The `void` transform may be used to replace the transform in an existing partiti #### Bucket Transform Details -Bucket partition transforms use a 32-bit hash of the source value(s). The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0. +Bucket partition transforms use a 32-bit hash of the source value. The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0. Transforms are parameterized by a number of buckets [1], `N`. The hash mod `N` must produce a positive value by first discarding the sign bit of the hash value. In pseudo-code, the function is: @@ -337,27 +337,11 @@ Transforms are parameterized by a number of buckets [1], `N`. The hash mod `N` m def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N ``` -When bucket transform is applied on a list of values, the hash is applied on the concatenated byte representations of all values. In pseudo-code, the function is: - -``` - def murmur3_x86_32_hashes(x1, x2, x3, ...) = { - byte[] bytes; - for (x in [x1, x2, x3, ...]) { - if (x != null) bytes.append(bytesOf(x)) - } - return murmur3_x86_32_hash(bytes) - } - - def bucket_N(x1, x2, x3, ...) = (murmur3_x86_32_hashes(x1, x2, x3, ...) & Integer.MAX_VALUE) % N -``` - Notes: 1. Changing the number of buckets as a table grows is possible by evolving the partition spec. -2. `murmur3_x86_32_hashes` produces the same result as `murmur3_x86_32_hash` when applied on a single value. -3. NULL input in the list of values is ignored when computing the hash. If all the input values are NULL, NULL should be produced. -For hash function details by type and bytes representation of each type, see Appendix B. +For hash function details by type, see Appendix B. #### Truncate Transform Details @@ -1076,27 +1060,6 @@ The types below are not currently valid for bucketing, and so are not hashed. Ho | **`float`** | `hashLong(doubleToLongBits(double(v))` [4]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` | | **`double`** | `hashLong(doubleToLongBits(v))` [4]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` | -For multiple arguments, `hashBytes()` is applied on the concatenated byte representation of each argument: - -| Primitive type | Bytes representation | -|----------------------|------------------------------------------------| -| **`int`** | `littleEndianBytes(long(v))` | -| **`long`** | `littleEndianBytes(v)` | -| **`decimal(P,S)`** | `minBigEndian(unscaled(v))` | -| **`date`** | `littleEndianBytes(daysFromUnixEpoch(v))` | -| **`time`** | `littleEndianBytes(microsecsFromMidnight(v))` | -| **`timestamp`** | `littleEndianBytes(microsecsFromUnixEpoch(v))` | -| **`timestamptz`** | `littleEndianBytes(microsecsFromUnixEpoch(v))` | -| **`timestamp_ns`** | `littleEndianBytes(nanosecsFromUnixEpoch(v))` | -| **`timestamptz_ns`** | `littleEndianBytes(nanosecsFromUnixEpoch(v))` | -| **`string`** | `utf8Bytes(v)` | -| **`uuid`** | `uuidBytes(v)` | -| **`fixed(L)`** | `v` | -| **`binary`** | `v` | - -For example, the hash representation of `(a:int, b:string)` will be `hashBytes(concatenation(littleEndianBytes(long(v)), utf8Bytes(b))`. - - Notes: 1. Integer and long hash results must be identical for all integer values. This ensures that schema evolution does not change bucket partition values if integer types are promoted. @@ -1160,14 +1123,12 @@ Each partition field in the fields list is stored as an object. See the table fo |--- |--- |--- | |**`identity`**|`JSON string: "identity"`|`"identity"`| |**`bucket[N]`**|`JSON string: "bucket[]"`|`"bucket[16]"`| -|**`bucket[N]`** (multi-arg bucket [1])| `JSON string: "bucketV2[]"` | `"bucketV2[16]"` | |**`truncate[W]`**|`JSON string: "truncate[]"`|`"truncate[20]"`| |**`year`**|`JSON string: "year"`|`"year"`| |**`month`**|`JSON string: "month"`|`"month"`| |**`day`**|`JSON string: "day"`|`"day"`| |**`hour`**|`JSON string: "hour"`|`"hour"`| -|**`Partition Field`** [2]|`JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}`|`{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}`| -|**`Partition Field with multi-arg transform`** [3]|`JSON object: {`
  `"source-id": -1,`
  `"source-ids": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}`|`{`
  `"source-id": -1,`
  `"source-ids": [1,2],`
  `"field-id": 1000,`
  `"name": "id_type_bucket",`
  `"transform": "bucketV2[16]"`
`}`| +|**`Partition Field`** [1,2]|`JSON object: {`
  `"source-id": ,`
  `"field-id": ,`
  `"name": ,`
  `"transform": `
`}`|`{`
  `"source-id": 1,`
  `"field-id": 1000,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`}`| In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec. @@ -1175,10 +1136,8 @@ The `field-id` property was added for each partition field in v2. In v1, the ref Notes: -1. For multi-arg bucket, the serialized form is `bucketV2[N]` instead of `bucket[N]` to distinguish it from the single-arg bucket transform. Therefore, old readers/writers will identify this transform as an unknown transform, old writer will stop writing the table if it encounters this transform, but old readers would still be able to read the table by scanning all the partitions. - This makes adding multi-arg transform a forward-compatible change, but not a backward-compatible change. -2. For partition fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted. -3. For partition fields with a transform of multiple arguments, the IDs of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. +1. For partition fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted. +2. For partition fields with a transform of multiple arguments, the IDs of the source fields are set on `source-ids`. To preserve backward compatibility, `source-id` is set to -1. ### Sort Orders @@ -1193,9 +1152,7 @@ Each sort field in the fields list is stored as an object with the following pro |Field|JSON representation|Example| |--- |--- |--- | -|**`Sort Field`** [1]|`JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| -|**`Sort Field with multi-arg transform`** [2]|`JSON object: {`
  `"transform": ,`
  `"source-id": -1,`
  `"source-ids": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucketV2[4]",`
  ` "source-id": -1,`
  ` "source-id": [1,2],`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| - +|**`Sort Field`** [1,2]|`JSON object: {`
  `"transform": ,`
  `"source-id": ,`
  `"direction": ,`
  `"null-order": `
`}`|`{`
  ` "transform": "bucket[4]",`
  ` "source-id": 3,`
  ` "direction": "desc",`
  ` "null-order": "nulls-last"`
`}`| Notes: 1. For sort fields with a transform with a single argument, the ID of the source field is set on `source-id`, and `source-ids` is omitted.