Introducing the React Native Nightly testing program 🌑🌓🌔🌕🌖🌗

[cortinico]

tl;dr: If you're a library maintainer, you can now sign up to be part of our nightly testing to make sure your library will keep on working

Hey all,
We’d like to introduce the React Native Nightly testing program:

react-native-community.github.io/nightly-tests

What is the Nightly testing program

Every night, a new version of react-native@nightly is published.

We then use that nightly version to run automated GitHub Actions workflows for testing some ecosystem libraries against that nightly build.

This allows us to catch breaking changes early on, and ensures that popular libraries keep on working across different React Native versions.

The jobs are running from this repository https://github.com/react-native-community/nightly-tests/ daily every night at 4:15 AM UTC using a GitHub Actions scheduled workflows.

The jobs essentially:

1. Create a new React Native app with the nightly version of React Native
2. Add the 3P library we intend to test
3. Run a build for both Android & iOS

The result of the build is reported in the result website as a 🟢 or 🔴 and let us monitor if libraries broke across the ecosystem:

How to join the Nightly testing program

We already have a list of libraries that we hand picked, but we’re looking into adding more libraries to expand our coverage.

At this stage we’re looking for libraries that are:

• Actively maintained (at least one release in the last year)
• Have more than 100 stars on GitHub
• Have more than 50.000 downloads on NPM
• Have native code (Java/Kotlin/C++/Swift/Objective-C) and are not JS-only libraries

If your libraries satisfies those requirements, please add your libraries by sending us a Pull Request that edits the libaries.json file as follows:

{ 
  "my-fancy-react-native-library": {
    "description": "A React Native library to display fancy animations",
    "installCommand": "my-fancy-react-native-library@nightly",
    "android": true,
    "ios": true,
    "maintainersUsernames": ["frodo", "bilbo", "pippin", "merry", "sam"],
    "notes": "Notes about the state of the library, like currently broken due to whatever reason"
  }
}

• description: This is a general description of the library just to know what the library does.
• installCommand: This is the command used by the yarn add <...> command
  • If your library publishes nightlies or betas, please use the @nightly and @beta suffix, so we’ll test against the latest version
  • If your library only publishes stable, just provide the npm package name here.
  • If your library has multiple NPM package that requires testing, please include all of them as space separated, and they will all be installed. For example: "installCommand": "react-native-reanimated@nightly react-native-worklets@nightly"
• android and ios: these are used to toggle if the test on Android or iOS should be executed
• maintainersUsernames: this is used to have an array of GitHub usernames. We will use it in the future to communicate with library maintainers (e.g. ping for build failures). Currently unused
• notes: these are notes about the state of the library, i.e. build currently broken due to a specific reason.

—

We hope this initiative will help us improve the stability of the ecosystem and ensure higher compatibility with future releases.

Nicola – for the release crew of React Native.

[ptoro]

Great to see the effort to improve coverage for third-party libraries!

I'm not a maintainer of this library, but I'd love to see react-native-firebase added. It meets the listed requirements and seems like a great fit because:

1. Changes in both nightly releases and stable releases have broken it before.
2. It's rather popular, so there's a large impact if it stops working.
3. It's Riccardo's favorite library! :) Joking aside, like he mentioned here, it's great for catching issues any new issues with the build system. It's a complex library, so if a change to the build system works with Firebase, the change is probably fine.

Side question for my own curiosity: is this related to the recent move of a lot of the nightly testing infrastructure to an RNC repo (i.e. to provide the CI headroom needed for testing additional third-party libraries)?

[cortinico]

@ptoro I'd suggest you reach out to the library maintainer and point them to include their library to our list

[kraenhansen]

From a quick glance it looks like it wouldn't be too hard for libraries to run the action themselves 🤞
I think it would be awesome if we could have this packaged as an action we can run on our own accounts / repos.

We tried doing something similar with our install tests at Realm, which ultimately ends up breaking as React Native moves along: The main benefit of having someone else maintain the action / workflow (as a library author) is lower likelihood of the workflow breaking because of a reason not related to the lib and then the workflow turning into a pile of unmaintained bit rot.

(We also chatted about this in person, but wanted to leave this as a comment for transparency too)

[cortinico]

I think it would be awesome if we could have this packaged as an action we can run on our own accounts / repos.

Yup we're open to this. Ideally we could export the action to a reusable workflow that libs could run. This is up for grab if someone wants to contribute and create a repo that we can them move under the https://github.com/react-native-community org

[hyochan]

Thank you! I am in ❤️

react-native-community/nightly-tests#19