Proposal: remove unpopular licenses

[wxiaoguang]

There are more than 720 licenses in Gitea https://github.com/go-gitea/gitea/tree/main/options/license

Most of them are out-dated/inactive/unpopular

We could just keep about 20-40 popular licenses:

• https://innovationgraph.github.com/global-metrics/licenses
• https://github.com/github/innovationgraph/blob/main/data/licenses.csv

Benefits:

• reduce binary size
• speed up execution time
• save memory
  • especially the License Detector, it consumes more than 100MB memory at the moment
  • by removing most unnecessary licenses, License Detector could only consume 3-5MB memory then

[delvh]

Another benefit:

• users are not overloaded with possible options anymore (I've often asked myself "which of these licenses could be the one I want")

[wxiaoguang]

So, as the first step, need to stop this:

@lunny @techknowlogick

[lunny]

So, as the first step, need to stop this:

@lunny @techknowlogick

A pull request could be sent to change cron-licenses.yml to only manually. #33486

[silverwind]

especially the License Detector, it consumes more than 100MB memory at the moment

100MB persistent? Excluding actual license data, it ideally should have close to zero persistent memory, only run when needed (e.g. when a repo was pushed to, does not matter if license update takes a few seconds in such cases, ideally it start off a lazy goroutine, debounced on like ~10s of repo push inactivity).

[silverwind]

20-40 popular licenses

How about keeping 50 most popular? Gitea's usersbase is diverse, 20 sounds definitely too low.

[eeyrjmr]

20-40 popular licenses

How about keeping 50 most popular? Gitea's usersbase is diverse, 20 sounds definitely too low.

maybe...
There only really appears to be ~30 that are really used on github,

import pandas as pd
url= "https://raw.githubusercontent.com/github/innovationgraph/refs/heads/main/data/licenses.csv"
tables= pd.read_csv(url)
tables[(tables.year == 2024)].pivot_table(values="num_pushers", index="spdx_license",aggfunc="sum").sort_values('num_pushers',ascending=False).plot(kind='barh',figsize=(9,9))

2024 data

2023 data

[silverwind]

If there is a reliable data source on license usage, could define a threshold of let's say >= 0.1% usage above which to include a certain license. I would commit the popularity source data into the repo with some instructions on how to update it.

[yp05327]

another benefit: we can remove the codes for detecting same license contents with similar license name. Actually, these codes are “magic codes”, it is better to remove them.
I think this is a good solution, the only one point is how we maintain these popular licenses list in the future. e.g. how to define the license is a popular license, are there any third part resources we can refer, are there any rules when the license is not popular any more.