CLN: spin msgpack support to [new] pandas-msgpack

[jreback]

similar to the split-off for pandas-gbq.

Would simplify the main pandas codebase a bit by making this a separately maintained package.

[jreback]

cc @llllllllll @ssanderson

[llllllllll]

Thanks for the heads up. We mainly use this with blaze so we just need to add that as a dependency to blaze[server]. I take it we would just need to import loads/dumps from pandas_msgpack instead of pandas, right?

[jreback]

@llllllllll I was just proposing this to see if people think its a good idea :>

but yes you could be explicit on the dependency.

my reasoning for proposing this are several fold really.

• its removes some code from main pandas and allows this package to have issues solved independently of the pandas release (not a big deal as this is pretty mature).
• this won't be deprecated in pandas 1.0, but I don't think will be supported in pandas 2 mainly because we are bringing on stream more 'standardized' formats (parquet / arrow / feather). These DO support wire formats (arrow), so that covers this usecase.
• the internal format of the msgpack is pretty pandas specific. not that this is a problem, but it makes interop with other systems a bit more tricky.

[jreback]

cc @wesm

[wesm]

+1 on splitting off this code

[wesm]

I also think splitting off pandas-json would be a good idea (I would like to consider deprecate that for a native RapidJSON-based C++ reader via Arrow sometime in the next year)

[ssanderson]

the internal format of the msgpack is pretty pandas specific. not that this is a problem, but it makes interop with other systems a bit more tricky.

This to me feels like an argument against splitting off this code into a separate package. If the pandas-msgpack format is tied to implementation details of pandas objects, then you would usually want to upgrade pandas-msgpack and pandas in lockstep, or else you risk breakage because your pandas-msgpack version is incompatible with your pandas version. Having the msgpack format in the pandas codebase itself ensures that you don't have to worry as much about version compatibility issues.

[wesm]

With continuous integration tools, we can automate the testing, so that doesn't seem like an issue. I think pandas's monolithic nature has made it harder for the community to make progress on components that may evolve at a different pace vs. the rest of the project.

[ssanderson]

I think pandas's monolithic nature has made it harder for the community to make progress on components that may evolve at a different pace vs. the rest of the project.

I agree with this, and as someone building an application on top of pandas, I appreciate the value of potentially slimming down the core distribution. At the same time, I suspect that the monolithic nature of pandas is actually a feature for many of pandas' users, since it means they don't have worry about or manage a constellation of pandas-* packages; they can just pip install pandas and get everything they might use.

One way to alleviate this concern might be to have something like a pandas-extra "meta-package", which would install the core pandas library along with related packages like pandas-msgpack. This would allow users to pick and choose which pieces they want if they care to do so, while still allowing users to install the whole world if they can't be bothered to figure out exactly which sub-packages they need.

[jreback]

actually metapackagea would be great for this

e.g. right now we should have a

pandas-perf which includes numexpr and bottleneck

you. could actually go nuts with this

pandas-io (excel, sql, hdf5, json)
pandas-io-extras (html, hig query, msgpsck)

so a big question is how to organize this to make it useful and not confusing

[jreback]

morning project!

https://github.com/pydata/pandas-msgpack
http://pandas-msgpack.readthedocs.io/en/latest/

not released yet.
is there an easy way to build wheels all at once (rather than using my separate envs to do this)? could/should have travis do it I suppose.