BUG: Fix regression for groupby.indices in case of unused categories#38649

jreback merged 7 commits intopandas-dev:masterpandas-dev/pandas:masterfrom

phofl:38642phofl/pandas:38642Copy head branch name to clipboard