adding automated releaser

[FoamyGuy]

These scripts were the result of combining some existing logic in adabot that checks if new releases are needed together with some library iteration code from Adabot and some pieces from scripts made by Eva for prior release roll outs.

A readme file is included that shows requirements and instructions on how to use the utility.

In summary this utility:

Uses git submodule foreach to loop over all libraries, uses Jinja2 to render a template for the release notes populated with appropriate values, then uses the GitHub CLI gh to make the release from the rendered release notes and a title pulled from a variable in one of the scripts.

The utility ensures there are newer commits than the latest release before acting. If there are not newer commits then the library will be skipped.

[tannewt]

Could you add a review step? I like making releasing as fast as possible but I want a human to review the unreleased changes and maybe outstanding issues. We may want to introduce some PR tags to note any breaking changes that should be carefully released.

[FoamyGuy]

@tannewt Initially I had the final "make_release()" call here: https://github.com/adafruit/adabot/pull/356/files#diff-34c53f9e30a2cb813b306e48039016f5b5f4b1d4d0050910ca9469f975a322a9R22
commented out so that when you run the script it does all of the checks and generates release notes for everything, but doesn't actually make the release.

Do you have a preferred way for the review mechanism to work? I could add a boolean that must be flipped to True for the final run in order to submit the release to github?

Also note that name of this might be prone to confusion. As it is set up now this tool must be run manually by a person. There are a few steps involved in cloning the bundle and then copying in these scripts and running them with the commands provided. When the person runs the tool it will iterate over the bundle and make the releases.

My intention is not for any of this to happen automatically as part of an actions run or cron job or anything like that. It still definitely requires a human to run it in order for the releases to be made. I don't really envision this being used as the primary way to make a release after "normal" PRs for individual libraries.

I mainly envision this being used specifically for cases when a patch change has been rolled out to all libraries and the release should ideally only include changes from that patch rollout, nothing from other PRs or unrelated changes.

[tannewt]

@tannewt Initially I had the final "make_release()" call here: https://github.com/adafruit/adabot/pull/356/files#diff-34c53f9e30a2cb813b306e48039016f5b5f4b1d4d0050910ca9469f975a322a9R22 commented out so that when you run the script it does all of the checks and generates release notes for everything, but doesn't actually make the release.

Do you have a preferred way for the review mechanism to work? I could add a boolean that must be flipped to True for the final run in order to submit the release to github?

I was thinking some sort of prompt that asks "here is the new version, the notes and the included changes" are you sure? If no, then leave release as draft. I'm thinking of a case where you'll want a major version number bump.

Also note that name of this might be prone to confusion. As it is set up now this tool must be run manually by a person. There are a few steps involved in cloning the bundle and then copying in these scripts and running them with the commands provided. When the person runs the tool it will iterate over the bundle and make the releases.

Ya, I get that.

My intention is not for any of this to happen automatically as part of an actions run or cron job or anything like that. It still definitely requires a human to run it in order for the releases to be made. I don't really envision this being used as the primary way to make a release after "normal" PRs for individual libraries.

I mainly envision this being used specifically for cases when a patch change has been rolled out to all libraries and the release should ideally only include changes from that patch rollout, nothing from other PRs or unrelated changes.

I know the patch cases is where you are coming from. I'm imagining this would be helpful for "release sweeps" where at some interval, we make sure libraries have been released. For patch cases, you could have a way to say "this commit is auto-yes".

[FoamyGuy]

The latest commit adds a prompt that serves two purposes:

1. adding a manual step to the process
2. allowing user to select patch, minor, or major version part to increase, or even skip the release for the library

I also added new logic and an additional prompt for the release title if one was not set inside of the configuration variable in make_release.py

These make the tool more a bit more general and allows it to cover a wider range of release use-cases.

Adding the input() for prompts also caused the way that the README.md file previously suggested enabling logging not to work so I updated the scripts to use the real logging module and by default it's configured to log to a file and to serial console.

Here is an example of the logged output and the prompts:

Entering 'libraries/helpers/progressbar'
2023-10-02 15:15:26,150 [INFO] Checking: libraries/helpers/progressbar
2023-10-02 15:15:26,151 [INFO] last commit: Mon Sep 18 21:18:11 2023
2023-10-02 15:15:26,469 [INFO] Latest release is: 2.3.12
2023-10-02 15:15:26,469 [INFO] createdAt: 2023-09-25T16:03:45Z
2023-10-02 15:15:26,471 [INFO] No new commits since last release, skipping
Entering 'libraries/helpers/pybadger'
2023-10-02 15:15:26,556 [INFO] Checking: libraries/helpers/pybadger
2023-10-02 15:15:26,558 [INFO] last commit: Tue Sep 26 17:25:17 2023
2023-10-02 15:15:26,887 [INFO] Latest release is: 3.7.13
2023-10-02 15:15:26,887 [INFO] createdAt: 2023-09-25T16:03:51Z
This library needs a new release. Please select a choice:
1. *default* Bump Patch, new tag would be: 3.7.14
2. Bump Minor, new tag would be: 3.8.0
3. Bump Major, new tag would be: 4.0.0
4. Skip releasing this library and go to next in the list
Choice, enter blank for default: 3
Selected '3'
2023-10-02 15:15:31,644 [INFO] Making a new release with tag: 4.0.0
Enter a Release Title: major bump

I've tested all four prompt menu options with scripts that were modified to just print the gh command instead of running it so that the behavior can be verified without making real releases, everything seems to be working as intended as far as I can tell.

@tannewt have another look when you get a chance. I'm not sure how to have it also output what the specific changes to the repo were. Let me know if I should continue digging into it to see if I can figure out how.

[tannewt]

I'm not sure how to have it also output what the specific changes to the repo were. Let me know if I should continue digging into it to see if I can figure out how.

The easiest thing would probably printing a link to the diff since the last release. Here is an example: adafruit/circuitpython-build-tools@1.10.4...main

[tannewt]

Thanks! The prompt looks good. One more question.

[FoamyGuy]

@tannewt the new commits have a few changes that make it possible to run this without the need to copy/paste any files

• Everything for releasing has been refactored into a single file adabot/circuitpython_library_release.py which can be executed with -m flag from the python interpreter.
• A pyproject.toml file has been added to make Adabot installable via pip. Currently there is nothing that would deploy it to pypi, so users would need to clone the adabot repo and then use pip install . inside of it to get it installed. I tested that process locally with success. If we also want to deploy to pypi we would need to set up a release action for it similar to libraries, we'd also need to start making releases when changes occur.
• Moved info about how to run it into the main README.rst file.

I ran this locally on the bundle and it did complete successfully, however it wasn't a complete test because there are currently no libraries in need of a release so it checked them all but never took me to a release prompt so that I could validate that portion of behavior.

I can try to test later tonight against a single library with a new commit in Adafruit_CircuitPython_TestRepo in order to confirm it's working as intended.

I'll also work later on making it output the changes link, that'll be easier to develop once I have a library that is actually needing the release to work with.

In my environment (py 3.10) I was unable to successfully install pyyaml at the version that was listed in requirements.txt There was an error about a type of license_file deprecation. I changed it to >= and found that I am able to install current version with just pip install pyyaml and then that allows the rest of pip install . to succeed. I've not tested the code that uses pyyaml though to ensure newer versions can work. It might be nice to un-pin some of the specific versions in requirements.txt if possible.

[tannewt]

• Everything for releasing has been refactored into a single file adabot/circuitpython_library_release.py which can be executed with -m flag from the python interpreter.

• A pyproject.toml file has been added to make Adabot installable via pip. Currently there is nothing that would deploy it to pypi, so users would need to clone the adabot repo and then use pip install . inside of it to get it installed. I tested that process locally with success. If we also want to deploy to pypi we would need to set up a release action for it similar to libraries, we'd also need to start making releases when changes occur.

This is exactly what I was thinking! Thanks! I don't think it needs to be on pypi but making it pip installable locally simplifies it a bunch.

Will look at the code now.

[tannewt]

One more small suggestion to make the cli a little cleaner

[FoamyGuy]

@tannewt The latest commit adds the adabot-release script alias thing (neat trick).

I've also added the compare link output:

My terminal links it, but it could be copy/pasted from a plain text only terminal.

Here is the full output after all prompts and it created the release successfully:

❯ adabot-release
2023-10-05 17:58:59,443 [INFO] Checking: repos/circuitpython/Adafruit_CircuitPython_TestRepo
2023-10-05 17:58:59,445 [INFO] last commit: Thu Oct 5 22:58:09 2023
2023-10-05 17:58:59,824 [INFO] 3.1.4
2023-10-05 17:58:59,824 [INFO] Latest release is: 3.1.4
2023-10-05 17:58:59,824 [INFO] createdAt: 2023-09-11T14:10:16Z
2023-10-05 17:59:00,111 [INFO] 3.1.4
This library needs a new release. Please select a choice:
Changes: https://github.com/adafruit/Adafruit_CircuitPython_TestRepo/compare/3.1.4...main
1. *default* Bump Patch, new tag would be: 3.1.5
2. Bump Minor, new tag would be: 3.2.0
3. Bump Major, new tag would be: 4.0.0
4. Skip releasing this library and go to next in the list
Choice, enter blank for default: 1
2023-10-05 18:02:06,879 [INFO] Making a new release with tag: 3.1.5
Enter a Release Title: Testing Adabot Release Tool
2023-10-05 18:02:16,309 [INFO] https://github.com/adafruit/Adafruit_CircuitPython_TestRepo/releases/tag/3.1.5

[tannewt]

Looks great! Thanks for the updates

[dhalbert]

@FoamyGuy is this all ready to merge?

[FoamyGuy]

@dhalbert I think it is ready. I do not have authorization to merge it.

It will definitely be helpful to have this to use after the patch from #360 which is merged and ready to go now as well.