Some codegate event names seem to be leaking guidance and not containing the correct summary

[wright-io]

Describe the issue

Some codegate events in the dashboard seem to have names that leak guidance as opposed to meaningful event names.

May be related to #875.

Steps to Reproduce

• Ask copilot with codegate to review https://github.com/stacklok/codegate-demonstration/blob/main/conf.ini (contains 3 secrets)
• Copilot responds "🛡️ CodeGate prevented 2 secrets from being leaked by redacting them"
• Dashboard contains some errors with names like "Please write a brief title for the chat conversation above. If the conversation covers multiple topics, you can just focus on the last one."

Operating System

MacOS (Arm)

IDE and Version

Extension and Version

Provider

OpenAI

Model

GPT-4o

Codegate version

v0.1.15

Logs

No response

Additional Context

No response

[lukehinds]

@danbarr this might be a docs desync

[danbarr]

@lukehinds I think this is two different issues. We know that we're not currently detecting AWS_SECRET_KEY because of the imprecise regex matching. I could just swap that out in the demo repo's conf.ini file to something we do detect (kind of hiding the issue vs. solving it, TBH).

The other issue @wright-io mentions is that we're logging Copilot's own prompt that it uses to generate the title for the conversation, but maybe this is intentional?

[wright-io]

@lukehinds Sorry - I think my description was unclear.

1. There are 3 secrets in the file but CodeGate is saying it detected only 2.
2. Although there are 3 secrets in the file, CodeGate is creating 5 alerts in response to the single prompt ("Review this file")
3. Some of the alerts seem to be logging the prompts CodeGate or Copilot might be generating for internal use (e.g. "Please write a brief title for the chat").

[danbarr]

#1 is because indeed, CodeGate only detected two. The regex for AWS secret keys is currently disabled; it was causing false-positives since there's no fixed format to those, it's just a random 40-character string and its length matches other strings like SHA hashes.

I had a thought about a regex that would add some context like looking for "aws" and/or "secret" on the same line - wouldn't be perfect but should detect most instances without false positives.

The other issues aren't docs/demo issues so I'll unassign myself @lukehinds

[aponcedeleonch]

Dan's ideas is the same as the one in #986

The issue of seeing more alerts than expected was fixed in #989

I will close this issue to only have one to track a way of dealing with false positive alerts on #986