Pandas time series plotting not working

[frederickayala]

Hello,

I created an Anaconda environment on MacOS using:
conda install jupyter pandas fbprophet seaborn

I have a Pandas DataFrame with a DatetimeIndex. I am plotting the time series using the pandas functionality:

df["ds"] = pd.to_datetime(df["ds"])
df = df.set_index("ds")
df[["y"]].plot()

The code works if I don't import fbprophet and fails after I import it with the error:
TypeError: float() argument must be a string or a number, not 'Period'

Checking the code I noticed that the matplotlib converters are being deregistered.

The code works if I add pd.plotting.register_matplotlib_converters() before plotting.

Is there a reason for using deregister_matplotlib_converters()?

Thanks

[robjellis]

+1, had the same issue. Thanks for this solution, worked perfectly for me!

[bletham]

Thanks for raising the issue.

This change was made to avoid a FutureWarning that was introduced around pandas 0.24. If you use fbprophet 0.4 with pd 0.24.2, then when you call plot on a Prophet model, you get

/usr/lib64/python2.7/site-packages/pandas/plotting/_converter.py:129: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.

To register the converters:
	>>> from pandas.plotting import register_matplotlib_converters
	>>> register_matplotlib_converters()
  warnings.warn(msg, FutureWarning)

Basically what's happening is that when you import pandas, it registers with matplotlib a bunch of functions that tell matplotlib how to plot pandas types. But, it does more than just that - it actually overwrites some built-in matplotlib handlers with pandas handlers. In particular, matplotlib has built-in the ability to plot datetime objects. When pandas is imported, it overwrites matplotlib's built-in datetime plotting with pandas datetime plotting.

Inside m.plot, we first convert everything out of pandas types before passing them along to matplotlib, specifically because we do not want to use pandas plotting. Despite that, in fbprophet v0.4 pandas was still being used for plotting because it overrode the datetime plotting. Thus plotting raised the FutureWarning despite us not using pandas types in plotting.

As noted in the FutureWarning, pandas is backtracking on the behavior and in the Future will require you to register the converters in order to plot pandas types in matplotlib. We thus decided to deregister them, to get the future behavior now.

The unfortunate thing here, though, is that it does break the behavior described in your post that previously worked, without any message as to why it doesn't work. In the future as more people upgrade to pd 0.24.2 and start to see the FutureWarning they will get in the habit of registering the converters prior to trying to plot their pandas types, but until then it will be a rocky transition. I mostly blame pandas for implementing this bad behavior and then backtracking on it in a not-backwards-compatible way, but perhaps there is something we could do to handle the transition better.

[frederickayala]

Thank you for the explanation 👍

[TomAugspurger]

Just FYI, we're tracking this in pandas over at pandas-dev/pandas#27036

Inside m.plot, we first convert everything out of pandas types before passing them along to matplotlib, specifically because we do not want to use pandas plotting. Despite that, in fbprophet v0.4 pandas was still being used for plotting because it overrode the datetime plotting. Thus plotting raised the FutureWarning despite us not using pandas types in plotting.

Would a setting like deregister_overridden_converters(), which preserves the (pandas-specific) Period converter, but restore the matplotlib converters for datetime, etc, work well for you?

[bletham]

@TomAugspurger I think that would do it.

[bletham]

Although if that function were added in a pandas version update then we'd still have to have some workaround for backwards compatability (we need to support at the least pd 0.23.4). Along those lines, I wonder if a clean solution might be in fbprophet instead of deregistering all, first check if pd has overriden the datetime plotter and then just change that one back if so.

[schmudde]

This seems to be common amongst notebook users, so I created a step-by-step recreation and explanation, in case it's helpful for others searching for this warning.

[nishifenger]

Thanks for this issue，it help me so much

[LL-Jan]

The solution works, hahaha

[TomAugspurger]

I think the best solution would be for prophet to issue all plot commands inside a context manager

@contextlib.contextmanager
def _pandas_converters():
    converters_active = check_if_active()  # something like pd.Timestamp in matplotlib.units.registry
    pd.plotting.deregister_matplotlib_converters()
    yield
    if converters_active:
        pd.plotting.register_matplotlib_converters()
    

And if you want to upstream that to pandas, it'd be great :)

[bletham]

@TomAugspurger that makes a lot of sense to me!

[ihadanny]

can you please fix the quick-start or at least mention this? https://facebook.github.io/prophet/docs/quick_start.html#python-api

[TomAugspurger]

FYI, this may be nicer in the next pandas release. Pandas is automatically registering the formatters just for the plot, and removing them when we're done. See https://dev.pandas.io/docs/user_guide/visualization.html#plotting-formatters Prophet may need to be updated to not set the option anymore (if it still is, I haven't checked), and just leave it as "auto".

…

On Tue, Nov 12, 2019 at 3:22 AM Ido Hadanny ***@***.***> wrote: can you please fix the quick-start or at least mention this? https://facebook.github.io/prophet/docs/quick_start.html#python-api — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#999?email_source=notifications&email_token=AAKAOIRVFDR3EQBTC52OLF3QTJYUVA5CNFSM4HSHMWUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDZSSYI#issuecomment-552806753>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIRFUS5TE5O2KIXFNWLQTJYUVANCNFSM4HSHMWUA> .