Migrate UnivariateFinite (for categorical distributions) out to new package

[ablaom]

In line with #416, I propose we move UnivariateFinite out to a new package called CategoricalDistributions.jl.

If this were okay with the current host of MLJBase.jl (I need to check this @vollmersj) it might make sense for this package to live at JuliaData (host of CategoricalArrays.jl) or JuliaStats (host of Distrtibutions.jl). I wonder what curators of those organisations think of that idea?

@nalimilan @bkamins @andreasnoack @devmotion @matbesancon

Recall that UnivariateFinite consists of the following:

• A composite type UnivariateFinite{S,V,R,P<:Real} for encoding the probability distribution associated with a finite labelled set of points, as opposed to the distribution Categorical from Distributions.jl, whose sample space is always a collection of integers. The sample space of a UnivariateFinite instance is a CategoricalPool object from CategoricalArrays.jl.

• Implementation of relevant parts of the Distributions.jl API, including rand, pdf, logpdf support, params, mode, and fit (which fits to a CatgoricalVector).

• A wrapper UnivariateFiniteArray for arrays of such objects (sharing a common sample space / pool). This type, implementing the AbstractArray API, is optimised for fast indexing, and for broadcasting of pdf, and logpdf (which turned out to be essential in our applications to machine learning).

• A fairly elaborate constructor for UnivariateFiniteArray objects from matrices of probabilities. See this docstring

Technical note. I'm hoping this migration should be fairly painless but there is one issue to be aware of: Currently the UnivariateFinite constructor stub lives in MLJModelInterface but the type and all real functionality lives in MLJBase (which depends on MLJModelInterface). The reason for this was to keep MLJModelInterface (the sole dependency of third party packages inplementing MLJ's model API) super lightweight. So this needs sorting out.

[ablaom]

cc @OkonSamuel

[DilumAluthge]

@cscherrer might also be interested in this.

[devmotion]

as opposed to the distribution Categorical from Distributions.jl, whose sample space is always a collection of integers.

While this is correct for Categorical, the support of the more general DiscreteNonParametric, of which Categorical is a special case, can be an arbitrary finite set of Reals.

[cscherrer]

Thanks @DilumAluthge . I guess it will get a little complicated since we're moving away from Distributions.jl, but I think we'll still be able to import this and wrap it as a Measure

[ablaom]

I guess it will get a little complicated since we're moving away from Distributions.jl,

Yes, Distributions is a hefty dependency. The new package could probably avoid Distributions as dependency and just implement the "functional" side of the Distributions API. Ie, drop the sub typing UnivariateFinite{...} <: Distribution or whatever. Unless Distributions wants to move types out to a new Base package - don't see that happening to be honest.

[OkonSamuel]

I guess it would be easier if Distributions.jl has a separate package containing basic functions which we can extend

[devmotion]

This was suggested and discussed in JuliaStats/Distributions.jl#1139 but it seems it was not supported by the maintainers.

[cscherrer]

Yes, Distributions is a hefty dependency.

Yes, but it's so widely used I don't see much getting around it. We actually have it as a dependency for MeasureTheory anyway, and use it for some of the sampling routines.

I'd love if this can be made nicely compatible with MeasureTheory, if it's not already. Here are some guidelines for that:

1. Structs themselves should not constrain types beyond what's necessary. For example, <: Real constraints impose a boundary for symbolic analysis that takes special care to get around. (edit by @ablaom: now fixed in MLJBase)
2. Any constructor with bounds checking should call to another function without it. The keyword-argument approach in Distributions.jl is awkward because it changes from one distribution to the next. (edit by @blaom: mostly fixed)
3. I'd love if there's a way to call this with different parameterizations. For example, log-weights or as input to the UnitVector transform from TransformVariables. Or the one from Bijectors, though honestly that one still confuses me since it's not bijective. We have a nice way to write parameterized measures in MeasureTheory.
4. If possible, any normalization constants should be computed separately (edit by @blaom: UnivariateFinite is now a general finite measure (unnormalised) so these don't appear, except in constructor when augment=true)

These are just off the top of my head, and partly for my notes as well.

[nalimilan]

CategoricalDistributions.jl really sounds like a narrow scope for a package. And IIUC it would define a lot of the Distributions.jl API (like pdf and such), and Distributions would have to depend on it (otherwise names would conflict)? At that point, we have to admit it would be very close to what a potential DistributionsBase package would be.

Why can't UnivariateFinite live in Distributions? Is it because you don't want to depend on Rmath?

[matbesancon]

I think this issue should be taken a bit broader than categorical distributions only, but my overall observation is that Distributions has become very hard to maintain because it is a central utility with few and rarely available maintainers. Most people just drop by for an issue or PR but don't want (understandably) to get involved in its maintenance, and many former maintainers are not involved anymore.

I agree with @nalimilan that splitting a package just for categorical distributions would be too fragmenting, but splitting a DistributionsBase, and possibly using this opportunity to clean the house, should be considered

[ablaom]

Why can't UnivariateFinite live in Distributions?

There were conversations but resistance. They said "you can just have Distributions as a dependency and overload..." I think the main objection was to include anything with a sample space that was not Array{Float64,N} or Array{Int,N} (curiously, probabilities have generic type, but the sample space types implemented are not generic).

[matbesancon]

include anything with a sample space that was not Array{Float64,N} or Array{Int,N}

(or Float64 / Int themselves). But overall I agree that this has been one of the biggest blocking points in Distributions, because some parts of the code base were non-trivial to adapt to something more generic. That being said, the type parameters of distributions can be extended (VariateForm and ValueSupport)

[ablaom]

but my overall observation is that Distributions has become very hard to maintain because it is a central utility with few and rarely available maintainers. Most people just drop by for an issue or PR but don't want (understandably) to get involved in its maintenance, and many former maintainers are not involved anymore.

This is also my impression and, if true, a real issue for Julia, in my opinion. The concept of a distribution extends far beyond what is in Distributions.jl (objects sampled using Monte-Carlo-Markov-Chain, pdfs on manifolds, priors for tree-like parameter spaces, etc). It may not make sense to add to what is there, but without a base API package it's very messy to extend.

[ablaom]

That being said, the type parameters of distributions can be extended (VariateForm and ValueSupport)

Yes, that is what we currently do in MLJ. But it means having Distributions.jl as a dependency in places we'd rather not. We had to perform some acrobatics to create our light-weight MLJModelInterface, partly for this reason.