BUG: pd.array([timedelta_like_strings]) should infer TimedeltaArray

[jbrockmendel]

>>> left = pd.array(['59 days', '59 days', pd.NaT]))
>>> left
<StringArray>
['59 days', '59 days', <NA>]
Length: 3, dtype: string

>>> left = pd.array(['59 days', '59 days', pd.NaT], dtype='m8[ns]')
>>> left
<TimedeltaArray>
['59 days', '59 days', NaT]
Length: 3, dtype: timedelta64[ns]

[jorisvandenbossche]

Why should it infer that? If passing strings., I think inferring a string type is best? And for conversions from string to timedelta64, we also have pd.to_timedelta

[TomAugspurger]

I think that'd be too magical. I think the result dtype of pd.array should only depend on the types of the values, not the values themselves.

[jbrockmendel]

1. pd.array should mirror pd.Series and pd.Index.
2. The presence of pd.NaT should be a big hint

[jorisvandenbossche]

I don't think we necessarily need to copy "mistakes" from Series/Index constructors into pd.array. Whether this case is one is debatable for sure.
But personally, I don't think we should automatically convert strings to datetimelike dtypes (without dtype being specified).

BTW, pd.array already does not fully mirror Series/Index, as it has the "future" behaviour fo defaulting to eg nullable integers.

[TomAugspurger]

Mmm I missed the NaT originally. That does change things, but I think it moves me to returning something like an object-dtype PandasArray.

What would your proposed behavior for something like

pd.array(['2 days', pd.NaT, 'a'])

be? A case where a value isn't valid for to_timedelta.

[jorisvandenbossche]

The presence of NaT could, besides the option to coerce to datetimelike as you propose, also be a reason to not infer string dtype though, if we want to differentiate different missing value sentinels.

This is actually also what happens with ints. You get object dtype instead of float or datetimelike:

In [5]: pd.Series([1, 2, pd.NaT]) 
Out[5]: 
0      1
1      2
2    NaT
dtype: object

EDIT: so yes what Tom also said ;)

[jorisvandenbossche]

If automatically coercing strings, the question also becomes which formats do automatically coerce and which not (apart from totally different strings as in Tom's example).

For example with datetime64:

In [10]: pd.Series(["2012/01/12", pd.NaT])    
Out[10]: 
0   2012-01-12
1          NaT
dtype: datetime64[ns]

In [11]: pd.Series(["01/12/2012", pd.NaT])    
Out[11]: 
0    01/12/2012
1           NaT
dtype: object

The one coerces, the other not, while both work fine in pd.to_datetime and result in the same datetime

[jbrockmendel]

I'm coming at this from a high-level view of "pd.array/pd.Index/pd.Series" should share code and behavior (#27460). If there is not consensus on that, then this topic is moot.

The "01/12/2012" example is a good one. That strikes me as surprising behavior in the Series constructor.

[jorisvandenbossche]

To be clear, I am fully on board that in the long term, those should share code (and basically Index/Series should just call pd.array under the hood, or something close to that).

But we could also deprecate the behaviour of inferring strings like that in Series/Index, in the course of unifying things.

[jbrockmendel]

But we could also deprecate the behaviour of inferring strings like that in Series/Index, in the course of unifying things.

Conditional on the behavior+code converging, i have no real objection to that.

[TomAugspurger]

+1 to consistency / sharing code. I think the only intentional difference is that array can return the new nullable types. For Series we'll need to wait for a global option / flag to the Series constructor to opt into that (and Index will need to wait for EA-backed indexes).

Are we agreed that the expected dtype for these mixed cases (strings and NaT) is object?

[jorisvandenbossche]

Are we agreed that the expected dtype for these mixed cases (strings and NaT) is object?

I think that is best.

An option could also be to raise an error (I don't know how easy it is code-wise), and requiring users to specify dtype=object explicitly if they want object dtype. In the idea that, for this specific case (with people passing pd.NaT), the result wil almost never be what you want, so better to notify people of it.
Although that is also a slippery slope (in which cases to raise, and in which to default to object dtype), so probably object dtype is best (if not coercing to datetimelike).

[jbrockmendel]

Are we agreed that the expected dtype for these mixed cases (strings and NaT) is object?

For the example with "a" in it, yes. For the datetime/timedelta-like strings, i'll need to give some thought to how id like Series/Index to behave longer-term.