From 6b68ce1a22ce7db2a5c114027cfa37aa38959018 Mon Sep 17 00:00:00 2001 From: Julien Le Dem Date: Mon, 15 Aug 2016 18:46:51 -0700 Subject: [PATCH 1/2] ARROW-252: Add implementation guidelines to the documentation --- format/Guidelines.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 format/Guidelines.md diff --git a/format/Guidelines.md b/format/Guidelines.md new file mode 100644 index 00000000000..136b3a20056 --- /dev/null +++ b/format/Guidelines.md @@ -0,0 +1,16 @@ +# Implementation guidelines + +An execution engine (or framework, or UDF executor, or storage engine, etc) can implements only a subset of the Arrow spec and/or extend it given the following constraints: + +## Implementing a subset the spec +### If only producing (and not consuming) arrow vectors. +any subset of the vector spec and the corresponding metadata can be implemented + +### If consuming and producing vectors +There is a minimal subset of vectors to be supported. +Production of a subset of vectors and their corresponding metadata is always fine. +Consumption of vectors should at least convert the unsupported input vectors to the supported subset (for example Timestamp.millis to timestamp.micros or int32 to int64) + +## Extensibility +An execution engine implementor can also extend their memory representation with their own vectors internally as long as they are never exposed. Before sending data to another system expecting Arrow data these custom vectors should be converted to a type that exist in the Arrow spec. +An example of this is operating on compressed data. From caf6994c782aa2f2e02e5817641193e24d39f4b8 Mon Sep 17 00:00:00 2001 From: Julien Le Dem Date: Sat, 20 Aug 2016 10:38:51 -0700 Subject: [PATCH 2/2] ARROW-252: review feedback --- format/Guidelines.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/format/Guidelines.md b/format/Guidelines.md index 136b3a20056..14f10578504 100644 --- a/format/Guidelines.md +++ b/format/Guidelines.md @@ -4,7 +4,7 @@ An execution engine (or framework, or UDF executor, or storage engine, etc) can ## Implementing a subset the spec ### If only producing (and not consuming) arrow vectors. -any subset of the vector spec and the corresponding metadata can be implemented +Any subset of the vector spec and the corresponding metadata can be implemented. ### If consuming and producing vectors There is a minimal subset of vectors to be supported. @@ -14,3 +14,4 @@ Consumption of vectors should at least convert the unsupported input vectors to ## Extensibility An execution engine implementor can also extend their memory representation with their own vectors internally as long as they are never exposed. Before sending data to another system expecting Arrow data these custom vectors should be converted to a type that exist in the Arrow spec. An example of this is operating on compressed data. +These custom vectors are not exchanged externaly and there is no support for custom metadata.