Skip to content

BUG: to_csv casting datetimes in categorical to int #44930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jreback merged 10 commits into from
Dec 22, 2021
Merged

BUG: to_csv casting datetimes in categorical to int #44930

jreback merged 10 commits into from
Dec 22, 2021

Conversation

@phofl
Copy link
Member

@phofl phofl commented Dec 16, 2021 •
edited
Loading

@phofl phofl added Categorical Categorical Data Type IO CSV read_csv, to_csv labels Dec 16, 2021
jbrockmendel
jbrockmendel reviewed Dec 16, 2021
View reviewed changes
@jbrockmendel
Copy link
Member

jbrockmendel commented Dec 16, 2021

i think the underlying problem is in Categorical.astype(object)

@phofl phofl marked this pull request as draft December 16, 2021 19:27
@phofl
Copy link
Member Author

phofl commented Dec 16, 2021

Was not sure if this was intentional in the astype call. Will have a look again

jbrockmendel
jbrockmendel reviewed Dec 17, 2021
View reviewed changes
jbrockmendel
jbrockmendel reviewed Dec 17, 2021
View reviewed changes
pandas/core/arrays/categorical.py Outdated
try:
new_cats = np.asarray(self.categories)
if is_datetime64_dtype(self.categories):
values = ensure_wrapped_if_datetimelike(np.asarray(self.categories))
Copy link
Member

@jbrockmendel jbrockmendel Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.categories._values should be what you want here. thats probably also an improvement if is_datetime64tz_dtype(self.categories.dtype)

Copy link
Member Author

@phofl phofl Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tz aware datetimes are alreday handled correctly

Copy link
Member

@jbrockmendel jbrockmendel Dec 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was thinking it might be a perf improvement for dt64tz, but either way its fine to consider that out of scope for this PR

jbrockmendel
jbrockmendel reviewed Dec 17, 2021
View reviewed changes
@phofl
Copy link
Member Author

phofl commented Dec 17, 2021

So I think I figured the issue out now.

Categorical.astype was incorrect for object dtype as you suggested. But this does not directly help us in the to_csv case, becaause this has to respect a date_format in to_native_types. We have to convert this to a DatetimeArray before doing the actual conversion.

@phofl phofl marked this pull request as ready for review December 17, 2021 14:34
jbrockmendel
jbrockmendel reviewed Dec 17, 2021
View reviewed changes
jbrockmendel
jbrockmendel reviewed Dec 17, 2021
View reviewed changes
jbrockmendel
jbrockmendel reviewed Dec 17, 2021
View reviewed changes
@jreback jreback added this to the 1.4 milestone Dec 17, 2021
jreback
jreback requested changes Dec 20, 2021
View reviewed changes
@jreback jreback merged commit 079289c into pandas-dev:master Dec 22, 2021
@jreback
Copy link
Contributor

jreback commented Dec 22, 2021

thanks @phofl

@phofl phofl deleted the 40754 branch December 22, 2021 09:00
@phofl phofl mentioned this pull request Dec 28, 2021
Merged
2 tasks
@simonjayhawkins simonjayhawkins mentioned this pull request May 9, 2022
Closed
3 tasks
@simonjayhawkins simonjayhawkins mentioned this pull request May 17, 2022
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jbrockmendel jbrockmendel jbrockmendel left review comments

@jreback jreback jreback approved these changes

Assignees

No one assigned

Labels

Categorical Categorical Data Type IO CSV read_csv, to_csv

Projects

None yet

Milestone

1.4

Development

Successfully merging this pull request may close these issues.

BUG: to_csv produces improper output for categorical datetimes .astype('object') misbehaves on Categorical containing Timestamp

3 participants

@phofl @jbrockmendel @jreback