Skip to content

Conversation

@thepatrickchin
Copy link
Contributor

@thepatrickchin thepatrickchin commented Sep 10, 2025

Description

This PR provides the frontend implementation corresponding to the Weave feedback changes in NVIDIA/NeMo-Agent-Toolkit#781. See also NVIDIA/NeMo-Agent-Toolkit#759 for additional context.

The feedback buttons trigger an API call that is available in NAT when Weave tracing is enabled. The endpoint expects a call ID that corresponds to the Weave trace for the specific agent message. The following summarizes how the call ID is extracted for different endpoints:

Weave Call ID Extraction by Endpoint Type

Non-Streaming Endpoints (/chat, /generate)

  • Backend → Proxy: Backend returns Weave-Call-Id in HTTP response header
  • Proxy → Frontend: Proxy gateway forwards the header unchanged
  • Frontend: Extracts from headers: response.headers.get('Weave-Call-Id')
  • Storage: Attached to the assistant message object in conversation history

Streaming Endpoints (/chat/stream, /generate/stream)

  • Backend → Proxy: Backend includes weave_call_id field in SSE data chunks (JSON payload)
  • Proxy → Frontend: Proxy extracts parsed.weave_call_id and wraps it in <weavecallid>ID</weavecallid> tags, injected into the stream
  • Frontend: Parses tags during stream processing with regex: /<weavecallid>([\s\S]*?)<\/weavecallid>/g and stores ID in extractedWeaveCallId variable
  • Storage: Attached to the assistant message object in conversation history

WebSocket Mode

  • Backend → Frontend: Backend includes weave_call_id field directly in WebSocket message payload
  • Frontend: Extracts weave_call_id from WebSocket message payload
  • Storage: Attached to the assistant message object in conversation history

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
    • Any contribution which contains commits that are not Signed-Off will not be accepted.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 10, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai
Copy link

coderabbitai bot commented Sep 10, 2025

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@thepatrickchin thepatrickchin force-pushed the weave-feedback branch 2 times, most recently from 2ef27bc to 1569eae Compare November 13, 2025 09:00
@thepatrickchin thepatrickchin changed the title Add feedback buttons to assistant messages that integrate with observability tools Add Weave-integrated feedback buttons to agent response messages Nov 13, 2025
@thepatrickchin thepatrickchin marked this pull request as ready for review November 13, 2025 09:44
@thepatrickchin thepatrickchin force-pushed the weave-feedback branch 2 times, most recently from 95be688 to fb4cf92 Compare December 17, 2025 03:44
Extract Weave-Call-Id from backend response and include it in response headers sent to client

Signed-off-by: Patrick Chin <[email protected]>
- Add event handler via hook
- Add /feedback endpoint to proxy's allowed list

Signed-off-by: Patrick Chin <[email protected]>
- code cleanup, remove unused code
- update related tests

Signed-off-by: Patrick Chin <[email protected]>
rapids-bot bot pushed a commit to NVIDIA/NeMo-Agent-Toolkit that referenced this pull request Dec 19, 2025
This PR enables user reaction feedback for AI assistant responses through Weave observability. Users just need to enable Weave observability – there is no additional configuration needed.

1. **Auto-detection**: NAT detects Weave telemetry config and automatically registers `/feedback` POST endpoint
2. **Call ID capture**: Root Weave call ID is stored in context when workflow starts
3. **Call ID propagation**: 
   - Non-streaming Endpoints: Call ID added to response headers
   - Streaming Endpoints: Call ID included in streaming response chunks
   - WebSocket Mode: Call ID included to WebSocket messages
5. **Frontend extraction**: UI extracts call ID from response and includes in conversation history
6. **Feedback submission**: Frontend calls `/feedback` with `weave_call_id` and `reaction_type`
7. **Weave integration**: Endpoint adds reaction directly to the corresponding Weave call

**See corresponding NAT-UI PR for frontend implementation NVIDIA/NeMo-Agent-Toolkit-UI#42**

Closes #759 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  - Any contribution which contains commits that are not Signed-Off will not be accepted.
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.



## Summary by CodeRabbit

* **New Features**
  * Per-run observability trace ID added to request context; propagated via HTTP header, SSE, and WebSocket with a dedicated observability-trace message (emitted during streaming and once after completion).
  * Conditional POST /feedback endpoint to record user reactions when Weave telemetry is enabled.

* **Documentation**
  * Docs added for the feedback endpoint, observability-trace message type, and Weave feedback integration.

* **Tests**
  * Tests for trace propagation, observability-trace messages, and /feedback behavior.

<sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub>

Authors:
  - Patrick Chin (https://github.com/thepatrickchin)
  - https://github.com/mnajafian-nv

Approvers:
  - Will Killian (https://github.com/willkill07)

URL: #781
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant