[Python] Provide mechanism for python to write out old format

[asfimport]

underlined textI think this needs to be an environment variable, so it can be made to work with old version of the Java library pyspark integration.

 

  @BryanCutler  can you check if this captures the requirements?

Reporter: Micah Kornfield / @emkornfield
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #5396

_{Note: This issue was originally created as ARROW-6474. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I would expose this as an option in the stream writer API. If Spark downstream releases are unable to apply patches to accommodate new Arrow releases I'm not sure how much we should bend over backwards to accommodate

[asfimport]

Bryan Cutler / @BryanCutler:
I think this is more for the case that a user is stuck with an already released Spark version, e.g. <= 2.4.4, and ends up installing pyarrow >= 0.15.0.  The pyarrow writers will use the new format by default, which the Arrow Java version in Spark will be unable to handle since it's using 0.14.1. There is no way for the user to set the option in the pyarrow writer either, so they would have to downgrade pyarrow. I think this it's fair to say they need to stick with pyarrow 0.14.1, but an env variable would give them a way to use the latest release.

[asfimport]

Wes McKinney / @wesm:
I see. I suppose the environment variable solution is not so awful.

I'll assign this issue to myself since we ought to have this functionality in Python before the 0.15.0 release

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 5396
#5396