ARROW-6314: [C++] Implement IPC message format alignment changes, provide backwards compatibility and "legacy" option to emit old message format

[wesm]

This also moves the alignment multiple to IpcOptions and adds the IpcOptions argument to more functions.

[wesm]

Will fix the CI issues and arrow_cuda failure (don't currently have CUDA installed on my laptop)

[emkornfield]

Took a quick scan through and seems reasonable. One question about if we want to memcpy for pre-0.15.0 version.

[pitrou]

LGTM overall.

[pitrou]

I pushed a fix for the CUDA issue.

[wesm]

@kszucs not sure what's up with the Buildbot builds

-- Could NOT find LLVM (missing: LLVM_DIR)
CMake Error at cmake_modules/FindLLVM.cmake:34 (find_package):
  Could not find a package configuration file provided by "LLVM" (requested
  version 7.1) with any of the following names:
    LLVMConfig.cmake
    llvm-config.cmake
  Add the installation prefix of "LLVM" to CMAKE_PREFIX_PATH or set
  "LLVM_DIR" to a directory containing one of the above files.  If "LLVM"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  src/gandiva/CMakeLists.txt:31 (find_package)

[kszucs]

@ursabot build

[wesm]

@emkornfield @tianchen92 note that Java is failing in a really ugly way when sent this new message format

Incompatible files
null
16:27:42.478 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.NullPointerException: null
	at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeDictionaryBatch(MessageSerializer.java:504)
	at org.apache.arrow.vector.ipc.ArrowFileReader.readDictionaryBatch(ArrowFileReader.java:178)
	at org.apache.arrow.vector.ipc.ArrowFileReader.readDictionary(ArrowFileReader.java:127)
	at org.apache.arrow.vector.ipc.ArrowFileReader.initialize(ArrowFileReader.java:110)
	at org.apache.arrow.vector.ipc.ArrowReader.ensureInitialized(ArrowReader.java:160)
	at org.apache.arrow.vector.ipc.ArrowReader.getVectorSchemaRoot(ArrowReader.java:62)
	at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:191)
	at org.apache.arrow.tools.Integration.run(Integration.java:118)
	at org.apache.arrow.tools.Integration.main(Integration.java:69)
	Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1024)
Allocator(ROOT) 0/1024/33428/2147483647 (res/actual/peak/limit)
		at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:454)
		at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:227)
		... 2 common frames omitted

I would guess that there are inadequate checks happening. For example it is only checked whether the metadata length is not equal to 0

https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L590

[wesm]

Travis CI succeeds except for integration tests, as expected: https://travis-ci.org/wesm/arrow/builds/577931299

Appveyor looking good: https://ci.appveyor.com/project/wesm/arrow/builds/27023793

Going to go ahead and merge this into the integration branch.

[wesm]

Merged into the integration branch