API Should Index be made opt-in?

[MarcoGorelli]

One of the points raised in pandas standardisation doc, in the context of the Consortium for Python Data API Standards, is about making indices opt-in

I've started putting together a write-up on this: https://hackmd.io/@mntOORP3TCesJJyvg-IdFQ/B1EqHfQfo

TLDR: the suggestion is to have a DefaultIndex which would always go from 0 to the length of the DataFrame, and then loc and iloc would be aligned. I think this'd be too big of a breaking change in pandas, and that there'd be a serious risk of silently ruining people's analyses/models. I'd be open to either introducing DefaultIndex but not have it be the default, or to make it the default but not give it a loc method

[phofl]

I agree, this is too big to change. Also, I am not convinced that this would be an improvement in general. This is what ignore_index was introduced for? This has lots of subtle implications in users code that introduce value dependent behavior.

Did you look into adding this as a metadata flag? Like with allow_duplicates? This would make it opt in and could be handled in something like __finalize__?

[rhshadrach]

The doc linked in the OP talks about certain libraries only supporting "default indexing". Could this be recast as libraries only supporting iloc? If one restricts usage of pandas to only using iloc, then pandas behaves as if it has "default indexing" today.

Are there other ops like loc that there is no iloc equivalent?

[phofl]

A couple that come to my mind:

• GroupBy is setting the group columns as index for example.
• concat takes the index into account when setting axis=1.

[rhshadrach]

Doh - of course; anything that uses alignment will immediately be incompatible. For groupby, I think one can just declare something like as_index=True is not supported when there is a default index. There will be edge cases here that break most certainly (probably in groupby.apply), but I think the most common cases will be okay.

If we created a default index object, I think we'd need to implement alignment for such objects via ilocs. For simplicity, let's say alignment between a default index and non-default index raises. If the internals use of alignment were sufficiently abstracted, maybe this could work, but I'd have to imagine that isn't the case currently and everything would just break.

Edit: And to make them sufficiently abstracted I imagine would be quite a perf hit.

[phofl]

Alignment is a good point. Did not think about this. I don't really see the benefit on our side or for our users here

[shwina]

If we created a default index object, I think we'd need to implement alignment for such objects via ilocs. For simplicity, let's say alignment between a default index and non-default index raises.

Perhaps it's helpful to think about this request as less of a DefaultIndex, and more of a NoIndex? Any situation that would normally require index alignment (e.g., Series([1, 2]) + Series([1, 2, 3])) would simply raise because of a mismatch in size.

Perhaps a config option would control whether objects are created with an index or not, so that you never have a situation where "default" and "nondefault" indexes coexist.

[MarcoGorelli]

Would the idea then be to raise if index_col is used in read_csv? And to raise for all the other options which involve indices (like as_index mentioned above)?

I'm worried this would add a lot of complexity, and I'm also not really clear what the benefit to users would be (though I'll concede that I've not attended an API Consortium meeting yet, and that maybe if I'd followed the whole discussion then I'd see it)

[shwina]

Thanks @MarcoGorelli and others! While I'm happy to continue discussing the technical details - which there are many to iron out - I feel the discussion would be greatly benefited by some additional context for this issue.

First, I should clarify that I agree on both points below:

• eliminating indexes entirely from Pandas is a non-starter in any reasonable time frame.
• Allowing Pandas users to opt-in (or even opt-out) of indexes is not immediately beneficial to them

However, I'm hoping we can convince Pandas maintainers to make Pandas "index unaware" by default (whatever that looks like), because it would benefit the ecosystem of DataFrame-like libraries and their users in the longer term.

The goal of the Consortium is to develop a "standard" DataFrame API, based on Pandas, that all Pandas-like Python libraries can support. This will enable users and downstream packages to switch out one library for another. A recent example of success of this idea from the Array world can be seen here.

Many Pandas-like libraries today either do not support indexes at all, or support a subset of index functionality. Dask, for example does not support MultiIndex. Speaking as a cuDF developer, we do support both Index and MultiIndex. But our code for doing so is clunky, and it's difficult/impossible to match Pandas' behaviour 100% of the time.

If we could drop indexes from our API and still match Pandas' default behaviour, we would do so in a heartbeat. I believe many other library maintainers feel the same way.

Opt-in v/s opt-out

The success of a "standard API" depends on matching the default behaviour of Pandas, which is why the request is framed as making Indexes opt-in, rather than opt-out.

[MarcoGorelli]

Thanks for sharing context! To clarify:

If we could drop indexes from our API and still match Pandas' default behaviour, we would do so in a heartbeat

would you then also drop .loc? Because my biggest issue is with the "proposal is to align loc and iloc" part

[shwina]

would you then also drop .loc? Because my biggest issue is with the "proposal is to align loc and iloc" part

Yes, the idea is that "standard API" would not include .loc, but Pandas could of course support a superset of the standard.

[MarcoGorelli]

OK thanks that makes more sense to me then - removing loc by default (rather than changing its behaviour to be aligned with iloc, which is what I'd originally understood this it be about) would be a safer breaking change

I'll rework the proposal in light of this

[jreback]

i am pretty skeptical here that you can change this w/o massive changes - nor do i think you should

but happy to see a sketch