ENH: allow plotting libraries to easily hook into pandas

[jtratner]

At PyData I was introduced to this great library that wraps matplotlib called prettyplotlib by @olgabot and it was used to good effect in @twiecki's presentation on PyMC3.

I want to incorporate this into pandas - it would be a great improvement and a light dep for the library.

@olgabot said she was interested in working on a PR to incorporate it, so going to track from this issue.

[jankatins]

prettyplolib also has a dependencies on brewer2mpl.

I'm not so sure about this: the lib changes mpl.rcParams directly (and some on import -> so simple importing prettyplotlib will change other unrelated matplotlib plots!) and this means that you can't use your own style anymore.

So, an example which I think won't work anymore is this:

import matplotlib.pyplot as plt
from pandas import *
from numpy.random import randn

ts = Series(randn(1000), index=date_range('1/1/2000', periods=1000))
ts = ts.cumsum()

# plots normal and if this is implemented "nicer" plots
ts.plot()
plt.show() 

# prints currently a xkcdified plot, but probably won't work if the plot function sets other 
# mpl.rcParams *after*  the `xkcd()` context manager sets the mpl.rcParams for the 
# xkcd version
with plt.xkcd():
    ts.plot()
    plt.show() 

There are also other libs like prettyplotlib out there, which wrap matplotlib to produce nicer plots:

• https://github.com/yhat/ggplot
• https://github.com/mwaskom/seaborn/

[EDIT: It would be nice if they have a way to us there own styling system without getting styles from whatever system you choose.]

There is also functions around, which style already existing matplotlib plots or set some "nicer" defaults:

• restyle a existing plot: https://github.com/wrobstory/climatic/blob/master/climatic/stylers.py -> rstyle(ax) -> http://nbviewer.ipython.org/urls/raw.github.com/wrobstory/climatic/master/examples/ggplot_styling_for_matplotlib.ipynb)
• saner mpl.rcParam defaults: https://github.com/mwaskom/seaborn/blob/master/seaborn/rcmod.py -> set()

[jtratner]

Okay, that's a very good point, let's keep it under an optional argument (that's disabled by default). We should be guarding mpl imports altogether so that matplotlib is not imported unless (and until) the user calls for plotting. Bottom line, I'd like pandas to have better plotting output because the default styles are not great and most people (including me) want to be able to not think about it.

Other notes:

• I suggest prettyplotlib because a number of people were using it at PyData, it looks nice, and the developer is both very engaged and was willing to do the work to integrate it.
• Yhat's ggplot is pretty much impossible because it's a completely different syntax and setup from matplotlib.
• Seaborn looks interesting; however it requires both moss and pyhusl, so it's not as if there are less deps.
• If it turns out to be too difficult to incorporate prettyplotlib directly, the stylers from climactic look like they could be a good option.

[jankatins]

If you "just" want to have a nice plotting:

import prettyplotlib
ts.plot()
plt.show()

(which shows that just importing prettyprintlib will change plotting for unrelated plots -> IMO a bug: olgabot/prettyplotlib#15 )

What could work is something like this (without any tests that it compiles...):

prettyplotlib_rcParams = {}
# hack, it would be nice to use a prettyplotlib.rc_context() -> see bugreport
with mpl.rc_context():
    import prettyplotlib
    prettyplotlib_rcParams.update(mpl.rcParams)
....

def _get_plotting_styles():
    name = [get the options system to spit out which styles should be used]
    if name in ["default", "prettyplotlib"]:
         return prettyplotlib_rcParams
    else:
         return {}

def plot(...):
    styles = _get_plotting_styles()
    with mpl.rc_context(rc=styles):
        [current ploting...]

[TomAugspurger]

Alternatively a settings in pd.set_option to declare globally that you want to use prettyplotlib's styling (or potentially others) may be useful.

I've been struggling with what pandas should be doing to support all the awesome plotting libraries sprouting up. (I'd add Bokeh and Vincent to the ones @JanSchulz listed). Providing the data structure and letting each library build off that is one thing. But the style of df.plot() is the tricky issue. I don't think something like df.plot(..., backend='prettyplotlib') would be ideal.

[jorisvandenbossche]

@jtratner How do you see it different to the current pd.options.display.mpl_style == "default"?
Maybe this style could be improved, or you could add a "prettyplotlib" style, so if you set that option prettyplotlib integration will be enabled?
(apart from this, default is a strange name, as this is not the default ..)

Regarding yhat's ggplot, this is not impossible, because, apart from the new syntax (which emulates matplotlib calls), it just uses some rcParams calls to define it's style (https://github.com/yhat/ggplot/blob/master/ggplot/themes/theme_gray.py), The "default" style could be updated to reflect the improvements here, or another style "ggplot" could be added based on these rcparams.

You also have mpltools where you can do something like style.use('style1'), which will be integrated in matplotlib in the future (matplotlib/matplotlib#2236).

[olgabot]

Hello, prettyplotlib developer here. I could also adjust the library so it
does everything programmatically and doesn't touch rcParams, and thus
other plotting issues. I agree that it's misleading that there's some
secret changes happening in the background, so thanks for opening that
issue. The pandas/tools/plotting.py file is pretty gnarly so I'm still
wrapping my head around it.

If there's a toggle like pd.options.display.mpl_style == "prettyplotlib"
then sounds like the plotting.py file should have a master dict or
something which matches plot and hist and bar and all that to the
different options.

---

Olga Botvinnik
PhD Program in Bioinformatics and Systems Biology
Gene Yeo Laboratory | Sanford Consortium for Regenerative Medicine
University of California, San Diego
olgabotvinnik.com
blog.olgabotvinnik.com
github.com/olgabot

On Tue, Nov 12, 2013 at 10:02 AM, Tom Augspurger
notifications@github.comwrote:

Alternatively a settings in pd.set_option to declare globally that you
want to use prettyplotlib's styling (or potentially others) may be
useful.

I've been struggling with what pandas should be doing to support all the
awesome plotting libraries sprouting up. (I'd add Bokeh and Vincent to the
ones @JanSchulz https://github.com/JanSchulz listed). Providing the
data structure and letting each library build off that is one thing. But
the style of df.plot() is the tricky issue. I don't think something like df.plot(...,
backend='prettyplotlib') would be ideal.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5489#issuecomment-28299975
.

[jankatins]

@jorisvandenbossche the theming in ggplots PR yhat/ggpy#75 will use more than mpl.rcParams to emulate ggplot2 theming: some things are not possible with just mpl styles. See the last commit in that PR.

[olgabot]

Either way with ggplot2 or prettyplotlib styling, I think the pandas
default should not use matplotlib defaults. This will definitely increase
the "wow" factor for first-time users too.

---

Olga Botvinnik
PhD Program in Bioinformatics and Systems Biology
Gene Yeo Laboratory | Sanford Consortium for Regenerative Medicine
University of California, San Diego
olgabotvinnik.com
blog.olgabotvinnik.com
github.com/olgabot

On Tue, Nov 12, 2013 at 10:20 AM, JanSchulz notifications@github.comwrote:

@jorisvandenbossche https://github.com/jorisvandenbossche the theming
in ggplots PR yhat/ggpy#75 https://github.com/yhat/ggplot/pull/75will use more than
mpl.rcParams to emulate ggplot2 theming: some things are not possible
with just mpl styles. See the last commit in that PR.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5489#issuecomment-28318726
.

I merged #3112 originally since the default mpl color theme is so horrible. However, It's off by
default because I considered changing another lib's settings as a side-effect of import pandas unacceptable, more so because the user may have her own customization in matplotlibrc and clobbering them without being asked is poor manners.

It may be possible to save/modify_rcParams/plot/restore to isolate pandas' from other matplotlib
clients, in which case I'm all for "pretty by default".

Note that the mpl_style option allows devs to define other styles, which you can leverage to supplement
the available styles. There's an optional callback mechanism triggered when an option is set you may find
useful.

pandas' deps have exploded in the last 6-9 months and increasingly support for multiple
deps providing similar functionality (html parsing, excel IO, serialization) is added. That has costs
in complexity, bug surface and on the test matrix but since it can take a long time for these costs
to mount there isn't much incentive for restraint except by admonition (apologies).

Would it be possible to integrate new styles by using the existing style mechanism rather then introducing
more deps?

[jtratner]

@y-p I agree with you here (and that's why I changed the title of this PR today). I want to make it easy to plug into pandas plotting. At the least, we should probably enable the mpl_style by default.

What I'd like to come out of this is a more abstracted plotting, maybe changing pandas/tools/plotting to be a set of static methods on a class that then gets looked up by NDFrame (and could then be subclassed by plotting libraries). Alternatively could just allow monkey patching.

@olgabot would it be enough to just have an init method that changed the default color scheme? (maybe monkey patching mpl_style). Or do you actually need to use prettyplotlib's methods?

[jankatins]

To make all mpl changes side effect free:

with mpl.rc_context():
    mpl.rcParams[...] = ...
    [or importing prettyplotlib or whatever...]
    [plotting commands as before]

There is also rc_context(rc=dict) to change the some mpl.rcParams styles directly, which could come from the option system...

[jtratner]

I think this may be geting off topic for this post, but it's very
reasonable to have plotting libraries provide functions that globally
change matplotlib (like some init function that you call explicitly or
something).

[olgabot]

The mpltools library does exactly that - it changes the rcParams (among
other things) to make all following plots the same. So if pandas plotting
is completely removed from rcParams, then they're unaffected. Any
thoughts? Should pandas plotting respond to matplotlib rcParams changes, or
should it stick to its own plotting?

FYI @y-p I'm having trouble installing matplotlib on my OS X Mavericks
machine which is why I haven't fixed your prettyplotlib issue yet.

---

Olga Botvinnik
PhD Program in Bioinformatics and Systems Biology
Gene Yeo Laboratory | Sanford Consortium for Regenerative Medicine
University of California, San Diego
olgabotvinnik.com
blog.olgabotvinnik.com
github.com/olgabot

On Thu, Nov 14, 2013 at 3:00 AM, Jeff Tratner notifications@github.comwrote:

I think this may be geting off topic for this post, but it's very
reasonable to have plotting libraries provide functions that globally
change matplotlib (like some init function that you call explicitly or
something).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5489#issuecomment-28475425
.