[ci] Add ids workflow for checking llvm apis have been annotated with LLVM_ABI

[mcbarton]

This PR adds a workflow which checks that llvm apis have been annotated with LLVM_ABI.

[llvmbot]

@llvm/pr-subscribers-llvm-adt

@llvm/pr-subscribers-github-workflow

Author: None (mcbarton)

Changes

This PR adds a workflow which checks that llvm apis have been annotated with LLVM_ABI.

---

Full diff: https://github.com/llvm/llvm-project/pull/128370.diff

1 Files Affected:

• (added) .github/workflows/ids-check.yml (+72)

diff --git a/.github/workflows/ids-check.yml b/.github/workflows/ids-check.yml
new file mode 100644
index 0000000000000..8ed12140d4bbc
--- /dev/null
+++ b/.github/workflows/ids-check.yml
@@ -0,0 +1,72 @@
+name: ids-check
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    name: ${{ matrix.name }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - name: ids-check
+            os: macos-15
+
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        repository: compnerd/ids
+        path: ${{ github.workspace }}/ids
+        fetch-depth: 0
+
+    - uses: actions/checkout@v4
+      with:
+        path: ${{ github.workspace }}/llvm-project
+        fetch-depth: 0
+
+    - name: Configure and build minimal LLVM for use by ids
+      run: |
+        cmake -B ${{ github.workspace }}/llvm-project/build/ \
+              -D CMAKE_BUILD_TYPE=Release \
+              -S ${{ github.workspace }}/llvm-project/llvm/ \
+              -D LLVM_ENABLE_PROJECTS=clang \
+              -D LLVM_TARGETS_TO_BUILD="host" \
+              -G Ninja
+        cd {{ github.workspace }}/llvm-project/build/
+        ninja -t targets all | grep "CommonTableGen: phony$" | grep -v "/" | sed 's/:.*//'
+
+    - name: install llvm and lit to build ids against
+      run: |
+        brew install llvm@18
+        brew install lit@18
+
+    - name: Configure ids
+      run: |
+        cmake -B ${{ github.workspace }}/ids/build/ \
+              -S ${{ github.workspace }}/ids/ \
+               -D CMAKE_BUILD_TYPE=Release \
+               -D CMAKE_C_COMPILER=$(brew --prefix llvm@18)/bin/clang \
+               -D CMAKE_CXX_COMPILER=$(brew --prefix llvm@18)/bin/clang++ \
+               -D CMAKE_EXE_LINKER_FLAGS=-fuse-ld=lld \
+               -D LLVM_DIR=$(brew --prefix llvm@18)lib/cmake/llvm \
+               -D Clang_DIR=$(brew --prefix llvm@18)/lib/cmake/clang \
+               -D FILECHECK_EXECUTABLE=$(brew --prefix llvm@18)/bin/FileCheck \
+               -D LIT_EXECUTABLE=$(brew --prefix lit@18)/bin/lit
+
+    - name: Build ids
+      run: |
+        cmake --build ${{ github.workspace }}/ids/build \
+              --config Release \
+              --parallel $(nproc --all)
+
+    - name: Run ids over compilation database
+      run: |
+        ${{ github.workspace }}/ids/build/bin/ids -export-macro LLVM_ABI -p ${{ github.workspace }}/llvm-project/build/compile_commands.json

[github-actions]

⚠️ We detected that you are using a GitHub private e-mail address to contribute to the repo.
Please turn off Keep my email addresses private setting in your account.
See LLVM Discourse for more information.

[tstellar]

How long does this take to run?

[mcbarton]

Hi @tstellar . The workflow I am trying to add takes around 5 minutes to run (with almost all of that being the checkout action). I created the workflow following a request by @compnerd and @vgvassilev in order to allow the work for the following issue to progress #109483 . Sorry for the large number of failed commits to try and get it to run. I'm trying to work out the exact idt command which should be run. Based on the test case added to ids here compnerd/ids#34, I feel I should have got the right command, but for some reason it doesn't work in the workflow.

[tstellar]

@mcbarton Can it run on the ubuntu runners? I don't think we have the capacity to run this on the macOS runners.

[mcbarton]

@mcbarton Can it run on the ubuntu runners? I don't think we have the capacity to run this on the macOS runners.

@tstellar Yes, I can modify the workflow to run on the Ubuntu runners.

[andrurogerz]

@mcbarton thanks for working on this! I took a quick look at the idt command in the job, and I see that you're not passing it the names of files to scan. In addition to passing it the comp database, I believe you also need to pass it the source files you specifically want it to scan as trailing arguments.

Something like this would scan all the files in the top commit:

idt -p ${{ github.workspace }}/llvm-project/build --export-macro=LLVM_ABI $(git diff --name-only HEAD~1 HEAD)

[mcbarton]

@andrurogerz Thanks for the suggestion. My workflow doesn't currently work with the suggestion since it cannot access the HEAD~1 commit (see https://github.com/llvm/llvm-project/actions/runs/13796002477/job/38587684995?pr=128370#step:8:5) . Do you have an alternative suggestion, or can you see the mistake in my workflow as to why it cannot access the commit?

[tstellar]

@andrurogerz Thanks for the suggestion. My workflow doesn't currently work with the suggestion since it cannot access the HEAD~1 commit (see https://github.com/llvm/llvm-project/actions/runs/13796002477/job/38587684995?pr=128370#step:8:5) . Do you have an alternative suggestion, or can you see the mistake in my workflow as to why it cannot access the commit?

You may want to look at the https://github.com/llvm/llvm-project/blob/main/.github/workflows/pr-code-format.yml#L21 workflow. I think it might do what you want.

[mcbarton]

@andrurogerz Thanks for the suggestion. My workflow doesn't currently work with the suggestion since it cannot access the HEAD~1 commit (see https://github.com/llvm/llvm-project/actions/runs/13796002477/job/38587684995?pr=128370#step:8:5) . Do you have an alternative suggestion, or can you see the mistake in my workflow as to why it cannot access the commit?

You may want to look at the https://github.com/llvm/llvm-project/blob/main/.github/workflows/pr-code-format.yml#L21 workflow. I think it might do what you want.

Thanks for the suggestion. I worked out my silly mistake after I sent that message. I wasn't in a git repo when I executed the command. The workflow is now running by based on https://github.com/llvm/llvm-project/actions/runs/13796145413/job/38588151792?pr=128370#step:8:6 . I just need to refine the diff so it only gives me the files with particular extensions.

[tstellar]

LGTM.

[boomanaiden154]

What exactly does this do and what are the success/fail criteria?

Do you have any idea of what the false positive and/or false negative rate looks like?

It would be good to have some documentation somewhere/leave a comment or something on failure with more information. Otherwise this is likely to just lead to warning fatigue.

[compnerd]

What exactly does this do and what are the success/fail criteria?

It simply scans the specified (assumed public) headers. The declarations in a public header should be annotated properly to indicate the ABI surface of the module. idt takes a set of parameters that indicates how to determine the CPP defines appropriate for the configuration to scan for public interfaces.

Do you have any idea of what the false positive and/or false negative rate looks like?

The false positive rate should be near zero. idt is pretty conservative in general when invoked properly (it does not know how to determine a header is private or not, and that is on the developer). Additionally, it does have the ability to exclude certain symbols from being scanned if needed.

[boomanaiden154]

The false positive rate should be near zero.

And it's clean when run over the entire monorepo already?

[mcbarton]

@boomanaiden154 I've addressed your suggestions. Can you take another look? Left it to you to set your reviews as resolved.

[boomanaiden154]

LGTM, assuming that the underlying check is green when run across the whole repo.

[vgvassilev]

Please don't merge this yet. There is some documentation coming soon explaining when this will be green, etc...

[compnerd]

The false positive rate should be near zero.

And it's clean when run over the entire monorepo already?

No it is not, and that is the reason for putting this up early. This is to help prevent backslide while the work continues to properly annotate the codebase. This is a long process and it isn't reasonable to put up a patch to make everything clean in one go as it would touch so many headers across so many different areas.

As we make progress, we can slowly increase the scanned area so that the signal from the run is helpful.

[boomanaiden154]

No it is not, and that is the reason for putting this up early. This is to help prevent backslide while the work continues to properly annotate the codebase. This is a long process and it isn't reasonable to put up a patch to make everything clean in one go as it would touch so many headers across so many different areas.

So if someone puts up a patch that touches a file that has not been annotated yet (completely unrelated to the annotation effort), this check would flag it?

[andrurogerz]

I have been using ids locally to keep up on new symbols while adding annotations throughout the codebase and it has been working well. I am generally running it on Windows, though the analysis should be the same running on Ubuntu. It might be worth switching this job to run on a Windows-runner if possible, though.

There are some additional things the job will need to consider to make it work properly:

1. Only run ids on changed header files under llvm/include/llvm. Exclude the subdir llvm/include/llvm/Debuginfod because Debuginfod is not an llvm component library. It should also exclude files under llvm/include/llvm/Demangle and run those separately with different command line args. I have found it works best to invoke ids on one file at a time rather than pass it a bunch of files at once.

2. Since the LLVM build providing the compile_commands.json may not have LLVM_ABI defined correctly, explicitly define it via ids' --extra-arg:

--extra-arg="-DLLVM_ABI=__attribute__((visibility(\"default\")))" --extra-arg="-Wno-macro-redefined"

3. Make sure the header defining LLVM_ABI gets added when needed:

--include-header="llvm/Support/Compiler.h"

4. Make sure fixits are emitted. I'm not sure how these options interact when running in a GitHub action. Maybe it can't actually edit the files for you, in which case we should not include --inplace.

--apply-fixits --inplace

5. Run an alternate command for modified files under llvm/include/llvm-c because there is a different annotation and header file.

--export-macro=LLVM_C_ABI --include-header="llvm-c/Visibility.h" --extra-arg="-DLLVM_C_ABI=__attribute__((visibility(\"default\")))"

6. Similarly, run an alternate command for modified files under llvm/include/llvm/Demangle.

--export-macro=DEMANGLE_ABI --include-header="llvm/Demangle/Visibility.h" --extra-arg="-DEMANGLE_ABI=__attribute__((visibility(\"default\")))"

7. Because ids analyses transitive includes, it may flag files that are not part of the PR. We should filter out these results if possible to reduce noise.

[vgvassilev]

@compnerd, @andrurogerz should we try to land that now?

[boomanaiden154]

This needs to be changed if you want the workflow to run on stacked PRs:

https://llvm.org/docs/CIBestPractices.html#ensuring-workflows-run-on-the-correct-events

[mcbarton]

I wasn't sure exactly what to put, so I put the same as what is in premerge.yaml (excluding closed) . If its not right, can you please make a suggested change and I will apply that?

[boomanaiden154]

You should include closed too in the list. Otherwise copying from premerge.yaml probably works.

I'm assuming the intention is that this works on the release branch too?

[mcbarton]

I have copied closed from premerge.yaml too.

@vgvassilev @andrurogerz @compnerd can correct me if I am wrong, but yes, I think this workflow is wanted for release branches too.

[compnerd]

Yeah, we would want them on release branches as well.

[nikic]

Can you share how it will look like if this job fails?

[compnerd]

We really should be building this on Windows. While ids will properly scan on Linux, it does require that you have configured the build for the host that you want to scan for - that is, we want to be building a windows binary. I do not know if the EULA for MSVC allows you to have the installation on Linux for that cross-compilation.

[nikic]

Why do we want to run on Windows or compile for a Windows host? The symbol visibility functionality also applies to Linux. I'd expect that running this on Linux is simpler and faster. (Running for any one target will miss conditionally compiled code, but that's true regardless of which one you pick.)

[boomanaiden154]

Running on Linux should be faster.

We're also capacity constrained on Windows, so running it there (especially on every PR) will put additional pressure there.

[compnerd]

The scanning occurs through the clang tooling, which will build as if it were building for Linux on Linux. To build for Windows on Linux, you need to ensure that you are setting up the cross-compilation appropriately: i.e. you configure and have available the Windows SDK, Microsoft C runtime, etc. If we can arrange for that, doing this build on Linux would be fine.

[nikic]

You still haven't explained why it is important to target Windows. What's the problem with building on Linux for Linux (apart from conditionally compiled code, which will always be missed in one direction or the other)?

[mcbarton]

This is not a comment either way with regards to which runner to use. I will implement the ids/idt check on whatever platform is chosen. This is just an update as to where the ids workflow stands for both, before I go on leave for the next 2 weeks. I have managed to get ids/idt to build on Linux. It takes around 2-3 minutes to build ids (see https://github.com/llvm/llvm-project/actions/runs/16962718956/job/48079138233). I tested to see whether the Linux based workflow would detect if a PR had a missing LLVM_ABI by removing one from a class, and it didn't detect it, so something is wrong with the flags currently chosen.

Despite trying all day I am yet to get a successful Windows build of ids on either the Windows x86 or arm runners. If I do manage to get it to work, then it will likely take 5-10 minutes for the build and check based on my progress so far (there seems to be great variance in the speed of the windows runners even of the same architecture).

[compnerd]

You still haven't explained why it is important to target Windows. What's the problem with building on Linux for Linux (apart from conditionally compiled code, which will always be missed in one direction or the other)?

The macro expansions are platform specific: Linux doesn't use __declspec(dllexport) and __declspec(dllimport) which have different semantics from __attribute__((__visibility__(...))). We would get close with just a Linux build, but it won't be exact. ids basically just parses the source code as if it were the target you specify and then attempts to identify the attributes on the interfaces.

[nikic]

Can you provide an example where ids will produce different output when targeting Linux and Windows?

[mcbarton]

This change will be reverted before the PR goes in. It has only been made for now as the workflow should fail when changes like this are made.