Skip to content

Commit ee8cb75

Browse files
LukasDrews97LukasDrews97
authored
Add probabilistic classification to hiclass #minor (#119)
1 parent 5ee9b59 commit ee8cb75

36 files changed

+6523
-1114
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,8 @@ instance/
248248
# Sphinx documentation
249249
docs/_build/
250250
doc/_build/
251+
docs/examples/trained_model.sav
252+
docs/source/auto_examples/
251253

252254
# PyBuilder
253255
target/

‎Pipfile

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,30 @@ name = "pypi"
66
[packages]
77
networkx = "*"
88
numpy = "*"
9-
scikit-learn = "*"
9+
scikit-learn = "1.4.2"
10+
scipy = "1.11.4"
1011

1112
[dev-packages]
12-
pytest = "*"
13-
pytest-flake8 = "*"
14-
pytest-pydocstyle = "*"
15-
pytest-cov = "*"
13+
pytest = "7.1.2"
14+
flake8 = "4.0.1"
15+
pytest-flake8 = "1.1.1"
16+
pydocstyle = "6.1.1"
17+
pytest-pydocstyle = "2.3.0"
18+
pytest-cov = "3.0.0"
1619
twine = "*"
17-
sphinx = "4.1.1"
18-
sphinx-rtd-theme = "0.5.2"
20+
sphinx = "5.0.0"
21+
sphinx-rtd-theme = "1.0.0"
22+
readthedocs-sphinx-search = "0.1.2"
23+
sphinx_code_tabs = "0.5.3"
24+
sphinx-gallery = "0.10.1"
25+
matplotlib = "3.9.2"
26+
pandas = "1.4.2"
27+
bert-sklearn = {git = "https://github.com/charles9n/bert-sklearn.git@master", editable = true}
28+
black = {version = "24.3.0", extras = ["colorama"]}
29+
pre-commit = "2.20.0"
30+
pyfakefs = "*"
31+
shap = "0.44.1"
32+
xarray = "2023.1.0"
1933

2034
[extras]
2135
ray = "*"

‎Pipfile.lock

Lines changed: 2313 additions & 1010 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/examples/plot_calibration.py

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# -*- coding: utf-8 -*-
2+
"""
3+
=====================
4+
Calibrating a Classifier
5+
=====================
6+
7+
A minimalist example showing how to calibrate a HiClass LCN model. The calibration method can be selected with the :literal:`calibration_method` parameter, for example:
8+
9+
.. tabs::
10+
11+
.. code-tab:: python
12+
:caption: Isotonic Regression
13+
14+
rf = RandomForestClassifier()
15+
classifier = LocalClassifierPerNode(
16+
local_classifier=rf,
17+
calibration_method='isotonic'
18+
)
19+
20+
.. code-tab:: python
21+
:caption: Platt scaling
22+
23+
rf = RandomForestClassifier()
24+
classifier = LocalClassifierPerNode(
25+
local_classifier=rf,
26+
calibration_method='platt'
27+
)
28+
29+
.. code-tab:: python
30+
:caption: Beta scaling
31+
32+
rf = RandomForestClassifier()
33+
classifier = LocalClassifierPerNode(
34+
local_classifier=rf,
35+
calibration_method='beta'
36+
)
37+
38+
.. code-tab:: python
39+
:caption: IVAP
40+
41+
rf = RandomForestClassifier()
42+
classifier = LocalClassifierPerNode(
43+
local_classifier=rf,
44+
calibration_method='ivap'
45+
)
46+
47+
.. code-tab:: python
48+
:caption: CVAP
49+
50+
rf = RandomForestClassifier()
51+
classifier = LocalClassifierPerNode(
52+
local_classifier=rf,
53+
calibration_method='cvap'
54+
)
55+
56+
Furthermore, probabilites of multiple levels can be aggregated by defining a probability combiner:
57+
58+
.. tabs::
59+
60+
.. code-tab:: python
61+
:caption: Multiply (Default)
62+
63+
rf = RandomForestClassifier()
64+
classifier = LocalClassifierPerNode(
65+
local_classifier=rf,
66+
calibration_method='isotonic',
67+
probability_combiner='multiply'
68+
)
69+
70+
.. code-tab:: python
71+
:caption: Geometric Mean
72+
73+
rf = RandomForestClassifier()
74+
classifier = LocalClassifierPerNode(
75+
local_classifier=rf,
76+
calibration_method='isotonic',
77+
probability_combiner='geometric'
78+
)
79+
80+
.. code-tab:: python
81+
:caption: Arithmetic Mean
82+
83+
rf = RandomForestClassifier()
84+
classifier = LocalClassifierPerNode(
85+
local_classifier=rf,
86+
calibration_method='isotonic',
87+
probability_combiner='arithmetic'
88+
)
89+
90+
.. code-tab:: python
91+
:caption: No Aggregation
92+
93+
rf = RandomForestClassifier()
94+
classifier = LocalClassifierPerNode(
95+
local_classifier=rf,
96+
calibration_method='isotonic',
97+
probability_combiner=None
98+
)
99+
100+
101+
A hierarchical classifier can be calibrated by calling calibrate on the model or by using a Pipeline:
102+
103+
.. tabs::
104+
105+
.. code-tab:: python
106+
:caption: Default
107+
108+
rf = RandomForestClassifier()
109+
classifier = LocalClassifierPerNode(
110+
local_classifier=rf,
111+
calibration_method='isotonic'
112+
)
113+
114+
classifier.fit(X_train, Y_train)
115+
classifier.calibrate(X_cal, Y_cal)
116+
classifier.predict_proba(X_test)
117+
118+
.. code-tab:: python
119+
:caption: Pipeline
120+
121+
from hiclass import Pipeline
122+
123+
rf = RandomForestClassifier()
124+
classifier = LocalClassifierPerNode(
125+
local_classifier=rf,
126+
calibration_method='isotonic'
127+
)
128+
129+
pipeline = Pipeline([
130+
('classifier', classifier),
131+
])
132+
133+
pipeline.fit(X_train, Y_train)
134+
pipeline.calibrate(X_cal, Y_cal)
135+
pipeline.predict_proba(X_test)
136+
137+
In the code below, isotonic regression is used to calibrate the model.
138+
139+
"""
140+
from sklearn.ensemble import RandomForestClassifier
141+
142+
from hiclass import LocalClassifierPerNode
143+
144+
# Define data
145+
X_train = [[1], [2], [3], [4]]
146+
X_test = [[4], [3], [2], [1]]
147+
X_cal = [[5], [6], [7], [8]]
148+
Y_train = [
149+
["Animal", "Mammal", "Sheep"],
150+
["Animal", "Mammal", "Cow"],
151+
["Animal", "Reptile", "Snake"],
152+
["Animal", "Reptile", "Lizard"],
153+
]
154+
155+
Y_cal = [
156+
["Animal", "Mammal", "Cow"],
157+
["Animal", "Mammal", "Sheep"],
158+
["Animal", "Reptile", "Lizard"],
159+
["Animal", "Reptile", "Snake"],
160+
]
161+
162+
# Use random forest classifiers for every node
163+
rf = RandomForestClassifier()
164+
165+
# Use local classifier per node with isotonic regression as calibration method
166+
classifier = LocalClassifierPerNode(
167+
local_classifier=rf, calibration_method="isotonic", probability_combiner="multiply"
168+
)
169+
170+
# Train local classifier per node
171+
classifier.fit(X_train, Y_train)
172+
173+
# Calibrate local classifier per node
174+
classifier.calibrate(X_cal, Y_cal)
175+
176+
# Predict probabilities
177+
probabilities = classifier.predict_proba(X_test)
178+
179+
# Print probabilities and labels for the last level
180+
print(classifier.classes_[2])
181+
print(probabilities)

‎docs/source/algorithms/calibration.rst

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
.. _calibration-overview:
2+
3+
===========================
4+
Classifier Calibration
5+
===========================
6+
HiClass provides support for probability calibration using various post-hoc calibration methods.
7+
8+
++++++++++++++++++++++++++
9+
Motivation
10+
++++++++++++++++++++++++++
11+
While many machine learning models can output uncertainty scores, these scores are known to be often poorly calibrated [1]_ [2]_. Model calibration aims to improve the quality of probabilistic forecasts by learning a transformation of the scores, using a separate dataset.
12+
13+
++++++++++++++++++++++++++
14+
Methods
15+
++++++++++++++++++++++++++
16+
17+
HiClass supports the following calibration methods:
18+
19+
* Isotonic Regression [3]_
20+
21+
* Platt Scaling [4]_
22+
23+
* Beta Calibration [5]_
24+
25+
* Inductive Venn-Abers Calibration [6]_
26+
27+
* Cross Venn-Abers Calibration [6]_
28+
29+
++++++++++++++++++++++++++
30+
Probability Aggregation
31+
++++++++++++++++++++++++++
32+
33+
Combining probabilities over multiple levels is another method to improve probabilistic forecasts. The following methods are supported:
34+
35+
Conditional Probability Aggregation (Multiply Aggregation)
36+
--------------
37+
Given a node hierarchy with :math:`n` levels, the probability of a node :math:`A_i`, where :math:`i` denotes the level, is calculated as:
38+
39+
:math:`\displaystyle{\mathbb{P}(A_1 \cap A_2 \cap \ldots \cap A_i) = \mathbb{P}(A_1) \cdot \mathbb{P}(A_2 \mid A_1) \cdot \mathbb{P}(A_3 \mid A_1 \cap A_2) \cdot \ldots}`
40+
:math:`\displaystyle{\cdot \mathbb{P}(A_i \mid A_1 \cap A_2 \cap \ldots \cap A_{i-1})}`
41+
42+
Arithmetic Mean Aggregation
43+
--------------
44+
:math:`\displaystyle{\mathbb{P}(A_i) = \frac{1}{i} \sum_{j=1}^{i} \mathbb{P}(A_{j})}`
45+
46+
Geometric Mean Aggregation
47+
--------------
48+
:math:`\displaystyle{\mathbb{P}(A_i) = \exp{\left(\frac{1}{i} \sum_{j=1}^{i} \ln \mathbb{P}(A_{j})\right)}}`
49+
50+
++++++++++++++++++++++++++
51+
Code sample
52+
++++++++++++++++++++++++++
53+
54+
.. code-block:: python
55+
56+
from sklearn.ensemble import RandomForestClassifier
57+
58+
from hiclass import LocalClassifierPerNode
59+
60+
# Define data
61+
X_train = [[1], [2], [3], [4]]
62+
X_test = [[4], [3], [2], [1]]
63+
X_cal = [[5], [6], [7], [8]]
64+
Y_train = [
65+
["Animal", "Mammal", "Sheep"],
66+
["Animal", "Mammal", "Cow"],
67+
["Animal", "Reptile", "Snake"],
68+
["Animal", "Reptile", "Lizard"],
69+
]
70+
71+
Y_cal = [
72+
["Animal", "Mammal", "Cow"],
73+
["Animal", "Mammal", "Sheep"],
74+
["Animal", "Reptile", "Lizard"],
75+
["Animal", "Reptile", "Snake"],
76+
]
77+
78+
# Use random forest classifiers for every node
79+
rf = RandomForestClassifier()
80+
81+
# Use local classifier per node with isotonic regression as calibration method
82+
classifier = LocalClassifierPerNode(
83+
local_classifier=rf, calibration_method="isotonic", probability_combiner="multiply"
84+
)
85+
86+
# Train local classifier per node
87+
classifier.fit(X_train, Y_train)
88+
89+
# Calibrate local classifier per node
90+
classifier.calibrate(X_cal, Y_cal)
91+
92+
# Predict probabilities
93+
probabilities = classifier.predict_proba(X_test)
94+
95+
# Print probabilities and labels for the last level
96+
print(classifier.classes_[2])
97+
print(probabilities)
98+
99+
.. [1] Niculescu-Mizil, Alexandru; Caruana, Rich (2005): Predicting good probabilities with supervised learning. In: Saso Dzeroski (Hg.): Proceedings of the 22nd international conference on Machine learning - ICML '05. the 22nd international conference. Bonn, Germany, 07.08.2005 - 11.08.2005. New York, New York, USA: ACM Press, S. 625-632.
100+
101+
.. [2] Chuan Guo; Geoff Pleiss; Yu Sun; Kilian Q. Weinberger (2017): On Calibration of Modern Neural Networks. In: Doina Precup und Yee Whye Teh (Hg.): Proceedings of the 34th International Conference on Machine Learning, Bd. 70: PMLR (Proceedings of Machine Learning Research), S. 1321-1330.
102+
103+
.. [3] Zadrozny, Bianca; Elkan, Charles (2002): Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: Association for Computing Machinery (KDD ’02), S. 694-699.
104+
105+
.. [4] Platt, John (2000): Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In: Adv. Large Margin Classif. 10.
106+
107+
.. [5] Kull, Meelis; Filho, Telmo Silva; Flach, Peter (2017): Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In: Aarti Singh und Jerry Zhu (Hg.): Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Bd. 54: PMLR (Proceedings of Machine Learning Research), S. 623-631.
108+
109+
.. [6] Vovk, Vladimir; Petej, Ivan; Fedorova, Valentina (2015): Large-scale probabilistic predictors with and without guarantees of validity. In: C. Cortes, N. Lawrence, D. Lee, M. Sugiyama und R. Garnett (Hg.): Advances in Neural Information Processing Systems, Bd. 28: Curran Associates, Inc.
110+

‎docs/source/algorithms/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,4 @@ HiClass provides implementations for the most popular machine learning models fo
1717
multi_label
1818
metrics
1919
explainer
20+
calibration

‎docs/source/algorithms/metrics.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,19 @@
33
Metrics
44
====================
55

6+
Classification Metrics
7+
--------------
8+
69
According to [1]_, the use of flat classification metrics might not be adequate to give enough insight of which algorithm is better at classifying hierarchical data. Hence, in HiClass we implemented the metrics of hierarchical precision (hP), hierarchical recall (hR) and hierarchical F-score (hF), which are extensions of the renowned metrics of precision, recall and F-score, but tailored to the hierarchical classification scenario. These hierarchical counterparts were initially proposed by [2]_, and are defined as follows:
710

811
:math:`\displaystyle{hP = \frac{\sum_i|\alpha_i\cap\beta_i|}{\sum_i|\alpha_i|}}`, :math:`\displaystyle{hR = \frac{\sum_i|\alpha_i\cap\beta_i|}{\sum_i|\beta_i|}}`, :math:`\displaystyle{hF = \frac{2 \times hP \times hR}{hP + hR}}`
912

1013
where :math:`\alpha_i` is the set consisting of the most specific classes predicted for test example :math:`i` and all their ancestor classes, while :math:`\beta_i` is the set containing the true most specific classes of test example :math:`i` and all their ancestors, with summations computed over all test examples.
1114

15+
Calibration Metrics
16+
--------------
17+
18+
1219
.. [1] Silla, C. N., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1), 31-72.
1320
1421
.. [2] Kiritchenko, S., Matwin, S., Nock, R., & Famili, A. F. (2006, June). Learning and evaluation in the presence of class hierarchies: Application to text categorization. In Conference of the Canadian Society for Computational Studies of Intelligence (pp. 395-406). Springer, Berlin, Heidelberg.

‎docs/source/conf.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
# -- Project information -----------------------------------------------------
2424

2525
project = "hiclass"
26-
copyright = "2022, Fabio Malcher Miranda, Niklas Köhnecke"
26+
copyright = "2024, Fabio Malcher Miranda, Niklas Köhnecke"
2727
author = "Fabio Malcher Miranda, Niklas Köhnecke"
2828

2929

0 commit comments

Comments
 (0)