-
Notifications
You must be signed in to change notification settings - Fork 44
Add Weave-integrated feedback buttons to agent response messages #42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2ef27bc to
1569eae
Compare
60411ad to
10036b7
Compare
95be688 to
fb4cf92
Compare
Extract Weave-Call-Id from backend response and include it in response headers sent to client Signed-off-by: Patrick Chin <[email protected]>
Signed-off-by: Patrick Chin <[email protected]>
- Add event handler via hook - Add /feedback endpoint to proxy's allowed list Signed-off-by: Patrick Chin <[email protected]>
Signed-off-by: Patrick Chin <[email protected]>
… endpoints Signed-off-by: Patrick Chin <[email protected]>
Signed-off-by: Patrick Chin <[email protected]>
Signed-off-by: Patrick Chin <[email protected]>
Signed-off-by: Patrick Chin <[email protected]>
Signed-off-by: Patrick Chin <[email protected]>
- code cleanup, remove unused code - update related tests Signed-off-by: Patrick Chin <[email protected]>
fb4cf92 to
55301aa
Compare
This PR enables user reaction feedback for AI assistant responses through Weave observability. Users just need to enable Weave observability – there is no additional configuration needed. 1. **Auto-detection**: NAT detects Weave telemetry config and automatically registers `/feedback` POST endpoint 2. **Call ID capture**: Root Weave call ID is stored in context when workflow starts 3. **Call ID propagation**: - Non-streaming Endpoints: Call ID added to response headers - Streaming Endpoints: Call ID included in streaming response chunks - WebSocket Mode: Call ID included to WebSocket messages 5. **Frontend extraction**: UI extracts call ID from response and includes in conversation history 6. **Feedback submission**: Frontend calls `/feedback` with `weave_call_id` and `reaction_type` 7. **Weave integration**: Endpoint adds reaction directly to the corresponding Weave call **See corresponding NAT-UI PR for frontend implementation NVIDIA/NeMo-Agent-Toolkit-UI#42** Closes #759 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. ## Summary by CodeRabbit * **New Features** * Per-run observability trace ID added to request context; propagated via HTTP header, SSE, and WebSocket with a dedicated observability-trace message (emitted during streaming and once after completion). * Conditional POST /feedback endpoint to record user reactions when Weave telemetry is enabled. * **Documentation** * Docs added for the feedback endpoint, observability-trace message type, and Weave feedback integration. * **Tests** * Tests for trace propagation, observability-trace messages, and /feedback behavior. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> Authors: - Patrick Chin (https://github.com/thepatrickchin) - https://github.com/mnajafian-nv Approvers: - Will Killian (https://github.com/willkill07) URL: #781
Description
This PR provides the frontend implementation corresponding to the Weave feedback changes in NVIDIA/NeMo-Agent-Toolkit#781. See also NVIDIA/NeMo-Agent-Toolkit#759 for additional context.
The feedback buttons trigger an API call that is available in NAT when Weave tracing is enabled. The endpoint expects a call ID that corresponds to the Weave trace for the specific agent message. The following summarizes how the call ID is extracted for different endpoints:
Weave Call ID Extraction by Endpoint Type
Non-Streaming Endpoints (
/chat,/generate)Weave-Call-Idin HTTP response headerresponse.headers.get('Weave-Call-Id')Streaming Endpoints (
/chat/stream,/generate/stream)weave_call_idfield in SSE data chunks (JSON payload)parsed.weave_call_idand wraps it in<weavecallid>ID</weavecallid>tags, injected into the stream/<weavecallid>([\s\S]*?)<\/weavecallid>/gand stores ID inextractedWeaveCallIdvariableWebSocket Mode
weave_call_idfield directly in WebSocket message payloadweave_call_idfrom WebSocket message payloadBy Submitting this PR I confirm: