ARROW-2145/ARROW-2157: [Python] Decimal conversion not working for NaN values #1610

cpcloud · 2018-02-14T23:43:39Z

No description provided.

cpcloud · 2018-02-15T14:51:47Z

Restarted the travis jobs, they seemed to be failing during apt-get

pitrou · 2018-02-15T16:22:46Z

cpp/src/arrow/python/builtin_convert.cc

Is this actually required? You are only using the decimal type below.

I might be misunderstanding the use case for OwnedRefNoGIL.

My understanding is that if we're Py_XDECREFing something and we do not hold then GIL then we need to acquire it.

The GIL is released in Cython before the function that instantiates this class is called (and isn't subsequently acquired in that function). Assuming my understanding is correct then we need to acquire the GIL before calling Py_XDECREF on this.

Are you suggesting that because the order of destruction here is guaranteed to be the reverse of initialization order that we don't need to care that decimal_module_ holds the GIL because we'll never decref decimal_module_ before we decref decimal_type_?

What I mean is that you don't care about keeping a reference to the decimal module if you only need to use the decimal type:

>>> D = __import__('decimal').Decimal >>> sys.modules['decimal'] = None >>> D(100) Decimal('100')

Oh, awesome. There are a few places that we have unnecessary refs to the decimal module. I'll open a JIRA to clean those up (and remove this one). Thanks for the review.

pitrou · 2018-02-15T16:25:41Z

cpp/src/arrow/python/builtin_convert.cc

This won't work with Python Decimal nans, right? Try decimal.Decimal('nan').

pitrou · 2018-02-15T16:26:11Z

cpp/src/arrow/python/python-test.cc

What is the policy for adding tests here rather than in pyarrow/tests? It seems writing tests in pure Python is generally easier :-)

It's funny you mention that. I agree, but I wanted to be able to step through C++ code in the CLion IDE so I wrote the test here.

cpcloud · 2018-02-15T22:11:43Z

python/pyarrow/tests/test_convert_builtin.py

Let me add Decimal('nan') to this list.

cpcloud · 2018-02-15T22:52:57Z

cpp/src/arrow/python/helpers.cc

@wesm @pitrou is there any reason we shouldn't make these static global variables, so we don't have to call these APIs all over the place and inside of loops?

IMO it's fine to make it a static global (or a singleton, etc.), as long as we don't want to support subinterpreters perhaps.

By the way the nested block above doesn't seem needed.

Yep, okay I'll refactor and make those static

Hm so it appears that string interning of module names doesn't play well with C++ globals. I'll leave these as they are for now.

Right, you probably can't use OwnedRef here since the destructor would trigger too late. But you could have a PyObject*. The only downside is that it would make the object eternal.

pitrou · 2018-02-15T23:03:58Z

cpp/src/arrow/python/helpers.h

Why the consts? I don't think it makes sense, and you're bound to do a lot of casts as soon as you call the Python C API.

Note PyObject is non-const pretty much by construction, as it has a reference count that can be mutated by any operation.

Ok, I was following convention in the file. I can adjust.

I've been removing all the instances of const PyObject* after having a compiler warning from an internal Py-API, so happy to see them all go

pitrou · 2018-02-15T23:06:59Z

cpp/src/arrow/python/helpers.cc

The last argument can simply be the empty string AFAIR.

pitrou · 2018-02-15T23:09:25Z

cpp/src/arrow/python/helpers.cc

You should call PyDecimal_Check before calling the method. Also you should check if the method raises. And it's better to use PyObject_IsTrue rather than compare against Py_True.

Yes, not sure how I missed that :)

cpcloud · 2018-02-16T19:01:54Z

@pitrou any more comments here?

pitrou

See comments below. Also it seems that the AppVeyor build has failed, though it may be unrelated?

pitrou · 2018-02-18T16:01:26Z

cpp/src/arrow/python/builtin_convert.cc

This member doesn't seem used actually. Were you planning to use it with PyObject_IsInstance instead of the costlier call to internal::PyDecimal_Check?

Yes, thanks. Will fix.

pitrou · 2018-02-18T16:02:17Z

cpp/src/arrow/python/builtin_convert.cc

Might be nice to add a comment motivating the algorithm here (why compare the absolute values but then memorize the original value)?

Actually, reconsidering this based on your comment this really should be just the max scale. Negative scale should contribute to precision only if it would increase precision. The goal here is to "cast the widest net", ie the max precision and max scale. Negative scale complicates things a tiny bit. I'll add some commentary.

pitrou · 2018-02-18T16:03:46Z

cpp/src/arrow/python/common.h

I'm not a C++ expert, so I'm curious why this is necessary?

Constructors are not inherited by default. I'm not actually using this, so it should cost nothing at runtime. If we ever wanted to construct one of these with a pointer as it's first argument we'd have to define it anyway. I can remove it if you'd like.

No problem with me. I had forgotten about non-inheritance of constructors.

pitrou · 2018-02-18T16:07:26Z

cpp/src/arrow/python/numpy_to_arrow.cc

Can PandasObjectIsNull return true on a Decimal instance?

No, I'll fix that.

pitrou · 2018-02-18T16:09:14Z

python/pyarrow/tests/test_convert_builtin.py

Also check that the Arrow type was inferred correctly?

Yep will do.

wesm · 2018-02-21T23:19:19Z

needs rebase

cpcloud · 2018-02-23T17:50:25Z

Closed in favor of #1651

pitrou reviewed Feb 15, 2018

View reviewed changes

cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link

Member

pitrou Feb 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work with Python Decimal nans, right? Try decimal.Decimal('nan').

pitrou reviewed Feb 15, 2018

View reviewed changes

cpcloud force-pushed the ARROW-2157 branch from f9a6c18 to 9dcb675 Compare February 15, 2018 20:50

cpcloud commented Feb 15, 2018

View reviewed changes

cpcloud force-pushed the ARROW-2157 branch from 9dcb675 to b93fea9 Compare February 15, 2018 22:50

cpcloud commented Feb 15, 2018

View reviewed changes

pitrou reviewed Feb 15, 2018

View reviewed changes

cpcloud force-pushed the ARROW-2157 branch from b93fea9 to 0cb36e5 Compare February 16, 2018 00:26

cpcloud force-pushed the ARROW-2157 branch 3 times, most recently from 40d9b52 to a910633 Compare February 18, 2018 01:58

pitrou requested changes Feb 18, 2018

View reviewed changes

cpcloud force-pushed the ARROW-2157 branch from a910633 to 5e23d56 Compare February 22, 2018 16:10

cpcloud added 3 commits February 22, 2018 08:10

ARROW-2145: [Python] Decimal conversion not working for NaN values

e1f762b

Test currently fails

eb2b678

Fix test

9476ae8

cpcloud force-pushed the ARROW-2157 branch from 5e23d56 to 9476ae8 Compare February 22, 2018 16:10

cpcloud closed this Feb 23, 2018

cpcloud deleted the ARROW-2157 branch February 23, 2018 17:50

This was referenced Sep 23, 2020

[C++] Clean up unnecessary decimal module refs #18130

Closed

[Python] Decimal conversion not working for NaN values #18112

Closed

ARROW-2145/ARROW-2157: [Python] Decimal conversion not working for NaN values #1610

ARROW-2145/ARROW-2157: [Python] Decimal conversion not working for NaN values #1610

Conversation

cpcloud commented Feb 14, 2018

Uh oh!

cpcloud commented Feb 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Feb 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cpcloud commented Feb 16, 2018

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Feb 15, 2018 •

edited

Loading