Copilot Integration tests can generate false positives for insight.stacklok.com links

[wright-io]

Describe the issue

In attempting to add a utm_source=codegate querystring to links generated to insight.stacklok.com for malicious packages, I was able to make changes that caused the integration tests to pass (e.g. Copilot User asks about a malicious package). But this did not result in the build package providing the new link format.

NOTE: [The issue with the utm_source not being added has been fixed](- #765) but this issue is to track a possible issue with the integration testing approach.

The updated copilot integration tests passed (the response https://www.insight.stacklok.com/report/pypi/invokehttp?utm_source=codegate) but when I built the container locally, it generates the warning about invokehttp, but the links don't contain the string.
CodeGate version: v0.1.13-25c2368-dev, so it seems like I'm building and running the right container.

In order to get the desired behavior in the built container, it seems that the generated insight link needed to be updated in output.py.
But, the integration tests passed without doing this (just by updating default.yaml to have the querystring added and changing the integraton test expectations, but the actual result in the codegate response wasn't correctly changed without adjusting output.py.

Related Issues

• Add utm_source=codegate to generated insight links #749
• Adding utm_source=codegate not working #765

Steps to Reproduce

1. Update prompts/default.yaml to change the format of the malicious package reporting example

For example:

"**Warning:** CodeGate detected one or more malicious, deprecated or archived packages."
  - Pkg 1: [https://www.insight.stacklok.com/report/ecosystem/package_name](https://www.insight.stacklok.com/report/ecosystem/package_name?utm_source=codegate)
  - Pkg 2: [https://www.insight.stacklok.com/report/ecosystem/package_name](https://www.insight.stacklok.com/report/ecosystem/package_name?utm_source=codegate)
  - ...

2. Update the copilot_malicious_package_question test case to expect the new format
3. Run the integration tests for the copilot provider. The tests will pass.
4. Build the container and run it locally
5. Configure copilot with codegate
6. Manually test (e.g. asking something like "Show me how to use invokehttp")
7. Confirm that the resulting insight link does not contain the changes (see more details in the description above)

Operating System

MacOS (Arm)

IDE and Version

n/a

Extension and Version

n/a

Provider

OpenAI

Model

GPT-40

Codegate version

v0.1.13-25c2368-dev

Logs

No response

Additional Context

No response

[rdimitrov]

I think we can close this one as completed. The integration test is checking for the new link (that has utm_source=codegate appended), so we should be okay 👍

[jhrozek]

Closing for now. Reopen if needed @wright-io