Support commenting on a PR when trigger is "issue_comment" - Action is not creating any comments on PR

[nilsfriess]

I was trying to setup a Github action that runs when I open a PR and add a comment with the content /benchmark.
Here's my workflow file:

name: Benchmark test

on:
  issue_comment:
    types: [created, edited, deleted]

jobs:
  run-benchmark:
    if: contains(github.event.comment.html_url, '/pull/') && contains(github.event.comment.body, '/benchmark')
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4      
    - name: Run benchmark
      run: |
        python3 gen_random_result.py | tee ${GITHUB_WORKSPACE}/output.json
    - name: Download previous benchmark data
      uses: actions/cache@v4
      with:
        path: ./cache
        key: ${{ runner.os }}-benchmark

    - name: Store new result
      uses: benchmark-action/github-action-benchmark@v1
      with:
        tool: customSmallerIsBetter
        output-file-path: output.json
        external-data-json-path: ./cache/benchmark-data.json
        github-token: ${{ secrets.GITHUB_TOKEN }}
        summary-always: true
        comment-always: true

The gen_random_result.py script does exactly what its name says, it just generates some random data of the form

[
 {
    "name": "Test benchmark",
    "unit": "Seconds",
    "value": "0.4937825918725717"
 }
]

The action runs as expected but it doesn't generate a comment on the PR with the results.
Is there something I'm missing?

[nilsfriess]

I think I found out what the problem is: Since I trigger the workflow on issue_comment, the "parent context" (don't know what to call it) is not a PR, so in this section here

github-action-benchmark/src/write.ts

Lines 247 to 249 in 55f6195

	return await (pr?.number
	? leavePRComment(repoMetadata.owner.login, repoMetadata.name, pr.number, body, commentId, token)
	: leaveCommitComment(repoMetadata.owner.login, repoMetadata.name, commitId, body, commentId, token));

the function leaveCommitComment will be called. Don't know how to fix it yet, but it shouldn't be too hard to get the parent PR from a comment.

[WorldSEnder]

Over at yew, we also have a setup which currently only comments on the commits themselves, not the PR.

The benchmarking is run on pull_request in this workflow. Important for this discussion is that we then use the generated artifacts in a workflow_run dependent workflow. This avoids giving out permissions and running the benchmarking workflow in the context of the pull request itself, instead using the second benchmark from the main branch to perform the tasks of reacting to and posting the results.

We thus have a similar issue. While it's possible to guide benchmark-action/github-action-benchmark to the correct commit with the ref option, there is no way to guide it to the correct issue number (although we already recover that as part of the workflow).

[gaby]

@nilsfriess You're missing permissions in your workflow:

permissions:
  # deployments permission to deploy GitHub pages website
  deployments: write
  # contents permission to update benchmark contents in gh-pages branch
  contents: write
  # allow posting comments to pull request
  pull-requests: write

[Mossaka]

We at runwasi hit the same issue as the action is not commenting on the PR after the the PR was created even though the actions were triggered a few times becasue the PR was updated: containerd/runwasi#810

[ktrz]

Hey @nilsfriess
Thanks for filing this issue. I edited the title to make it more obvious what the expected fix is.

[ktrz]

hey @WorldSEnder
Your issue is a slightly different one. As I understand it in your case you are running the benchmarking action based on the completion of another workflow, right? In that case there is definitely no way to automatically infer what the relevant PR number is. So the solution could be to add an additional workflow input ie. pr-number to manually tell the action where to post. I'll create a separate issue for that --> #302

[ktrz]

@Mossaka
I don't understand the issue in your case. The action doesn't post a new comment for each update of a PR. It rather edits the existing comment to reflect the most recent changes