ARROW-9518: [Python] Deprecate pyarrow serialization #8255

jorisvandenbossche · 2020-09-24T11:50:36Z

No description provided.

jorisvandenbossche · 2020-09-24T11:57:58Z

WIP, still need to add a few additional deprecations, and add do an update of the docs explaining the deprecation with alternatives.

cc @rok @mrkn @dhirschfeld (since you contributed to serialization somewhat recently)

github-actions · 2020-09-24T12:08:15Z

https://issues.apache.org/jira/browse/ARROW-9518

dhirschfeld · 2020-09-24T23:20:18Z

Thanks for the ping. I was using that functionality. So I'm interested to see how I can replace it.

My goal is to pass objects (mapping of arrays, dataframes and primitive types) between languages such as Python, R and TypeScript. pickle, being python-only, is no good for this. I understand the previous functionality was also python only so I suppose I haven't lost anything there.

I'm still unsure if arrow is capable of this language-agnostic interop for blobs of heterogeneous types?

jorisvandenbossche · 2020-09-25T11:32:21Z

I was using that functionality. So I'm interested to see how I can replace it.

And so I am interested to understand how you were using it, to know what to write about a replacement

You mention the goal of passing objects to other languages, but since that is also not possible now with serialize: what do you use serialize for currently ?
Is it a use case where you can basically replace it with pickle?

I'm still unsure if arrow is capable of this language-agnostic interop for blobs of heterogeneous types?

At the moment, not for "random" objects, but only for types that fit into one of the serialization format / IPC message types. The pyarrow.serialization module always has been python specific. Expanding the IPC messages types might be possible, but is something that needs to be discussed on the dev mailing list to see if there is interest.

pitrou · 2020-10-07T09:57:07Z

python/pyarrow/__init__.py

+                "removed in a future version. Use pickle or the pyarrow IPC "
+                "functionality instead.")
+        if name == "SerializationContext":
+            _warnings.warn(_msg.format(name), DeprecationWarning)


I think we use FutureWarning instead of DeprecationWarning? (because the latter is silent by default)

pitrou · 2020-10-07T09:58:51Z

python/pyarrow/serialization.py

+        "removed in a future version. Use pickle or the pyarrow IPC "
+        "functionality instead.",
+        DeprecationWarning, stacklevel=2
+    )


Do you want to write a helper instead of pasting this snippet everywhere? :-)

Also, there are existing helpers in pyarrow.util

pitrou · 2020-10-07T09:59:55Z

python/pyarrow/tests/test_serialization.py

@@ -52,6 +52,9 @@
    sparse = None


+pytestmark = pytest.mark.filterwarnings("ignore:'pyarrow:DeprecationWarning")


Hmm... is this another pytest magic? What does it do exactly? Filter these warnings only for this test module?

Indeed, this is a way to suppress (ignore) all DeprecationWarnings that start with "pyarrow" in this test module. That's easier than catching all individual warnings, since this full file is about serialization and thus deprecated.
(and therefore I added a separate file to explicitly test the warnings are raised)

Will add a short comment about it

dhirschfeld · 2020-10-07T10:19:40Z

what do you use serialize for currently?

I've just got a proof-of-concept arrow serialization framework which can serialize arbitrary Python objects (inheriting from a base class). Unfortunately, after implementing that I found it's not language-agnostic so it's languished as a bit of a curiosity.

I need a language-agnostic serialization format which can serialise a Python Dict[str, object] where object is itself serializable to a Dict[str, primitive_type]. I'm performance sensitive and may need to serialize large arrays/DataFrames ("primitive" data-science types). Given that, I hoped pyarrow might fit the bill but it doesn't seem suited to serializing heterogeneous mappings?

Currently wondering if msgpack might be better suited but interested to hear if this is something on the radar/roadmap for arrow?

pitrou · 2020-10-07T10:22:57Z

pickle is the best thing available for arbitrary Python types and heteregenous data, IMO.

dhirschfeld · 2020-10-07T10:26:41Z

pickle is the best thing available for arbitrary Python types and heterogenous data, IMO.

Yep, but I also want to be able to read the serialized data in from R, TypeScript and C#

pitrou · 2020-10-07T10:27:30Z

Then you'll have to invent your own serialization format, or find another existing one.

dhirschfeld · 2020-10-07T10:39:06Z

Yeah, that's what I figured 😞. I've previously invented my own (with protocol buffers) but it's a big job so I was hoping to leverage off existing efforts. Will have to do some more research to see what other options there are...

jorisvandenbossche · 2020-10-07T13:21:30Z

I think we use FutureWarning instead of DeprecationWarning? (because the latter is silent by default)

I have a slight preference for using DeprecationWarning here, at first. Reasoning: DeprecationWarning is still visible if you are using the functionality directly (eg if you call pa.serialize() yourself in an interactive environment or a main script, you will see it by default, since Python 3.7 or so), but not visible if you are using some library that is using the deprecated functionality. This gives developers of libraries using this functionality the chance to fix those warnings, before their users see warnings they cannot do anything about.

My idea was then to bump it from DeprecationWarning to FutureWarning in the next release, for example.

pitrou · 2020-10-07T13:23:31Z

My idea was then to bump it from DeprecationWarning to FutureWarning in the next release, for example.

Then we should make sure to have a JIRA for this, otherwise we'll probably forget :)

pitrou · 2020-10-07T17:33:57Z

This needs rebasing now :-)

jorisvandenbossche · 2020-10-09T08:28:07Z

Rebased this

kszucs · 2020-10-09T09:42:23Z

python/pyarrow/tests/test_serialization_deprecated.py

+
+
+def test_serialization_deprecated():
+


kszucs · 2020-10-09T09:42:46Z

python/pyarrow/__init__.py

+# deprecated top-level access
+

 from pyarrow.filesystem import FileSystem as _FileSystem, LocalFileSystem as _LocalFileSystem


I think we have flake8 disabled on this file.

yeah, that's from the other PR, will fix

kszucs

LGTM, just few nits.

kszucs · 2020-10-09T09:54:40Z

Why are the tests in a separate file?

jorisvandenbossche · 2020-10-09T10:07:32Z

Why are the tests in a separate file?

Because that allows me to simply ignore all deprecation warnings in the actual test file (see the comment at the pytestmark there)

kszucs · 2020-10-09T10:27:33Z

Thanks Joris, merging!

jorisvandenbossche force-pushed the ARROW-9518-deprecate-serialize branch from aef7bc6 to 1ede379 Compare September 30, 2020 15:02

jorisvandenbossche requested a review from kszucs September 30, 2020 15:02

jorisvandenbossche marked this pull request as ready for review September 30, 2020 15:02

jorisvandenbossche force-pushed the ARROW-9518-deprecate-serialize branch from 1ede379 to 733ff51 Compare October 7, 2020 08:41

pitrou reviewed Oct 7, 2020

View reviewed changes

jorisvandenbossche added 4 commits October 9, 2020 09:26

ARROW-9518: [Python] Deprecate pyarrow serialization

de3ca87

more deprecations + tests + basic docs

7f2ce90

fixup tests for older python

67a0f67

use helper function to deduplicate warnings

17b0b19

jorisvandenbossche force-pushed the ARROW-9518-deprecate-serialize branch from 7fd922e to 17b0b19 Compare October 9, 2020 07:36

fixup

e8c4941

kszucs reviewed Oct 9, 2020

View reviewed changes

python/pyarrow/tests/test_serialization_deprecated.py Outdated

def test_serialization_deprecated():

Copy link

Member

kszucs Oct 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace

kszucs reviewed Oct 9, 2020

View reviewed changes

kszucs approved these changes Oct 9, 2020

View reviewed changes

lint

1783764

kszucs closed this in 599b458 Oct 9, 2020

jorisvandenbossche deleted the ARROW-9518-deprecate-serialize branch October 9, 2020 11:04

jorisvandenbossche mentioned this pull request Oct 10, 2022

ARROW-17863: [Python] Deprecate Plasma Python bindings #14343

Merged

asfimport mentioned this pull request Oct 9, 2020

[Python] Deprecate pyarrow serialization #25586

Closed

		@@ -52,6 +52,9 @@
		sparse = None


		pytestmark = pytest.mark.filterwarnings("ignore:'pyarrow:DeprecationWarning")

		# deprecated top-level access


		from pyarrow.filesystem import FileSystem as _FileSystem, LocalFileSystem as _LocalFileSystem

ARROW-9518: [Python] Deprecate pyarrow serialization #8255

ARROW-9518: [Python] Deprecate pyarrow serialization #8255

Uh oh!

Conversation

jorisvandenbossche commented Sep 24, 2020

Uh oh!

jorisvandenbossche commented Sep 24, 2020

Uh oh!

github-actions bot commented Sep 24, 2020

Uh oh!

dhirschfeld commented Sep 24, 2020

Uh oh!

jorisvandenbossche commented Sep 25, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dhirschfeld commented Oct 7, 2020

Uh oh!

pitrou commented Oct 7, 2020

Uh oh!

dhirschfeld commented Oct 7, 2020

Uh oh!

pitrou commented Oct 7, 2020

Uh oh!

dhirschfeld commented Oct 7, 2020

Uh oh!

jorisvandenbossche commented Oct 7, 2020

Uh oh!

pitrou commented Oct 7, 2020

Uh oh!

pitrou commented Oct 7, 2020

Uh oh!

jorisvandenbossche commented Oct 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kszucs left a comment

Choose a reason for hiding this comment

Uh oh!

kszucs commented Oct 9, 2020

Uh oh!

jorisvandenbossche commented Oct 9, 2020

Uh oh!

kszucs commented Oct 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants