Skip to content

Introduce evaluators for agentic workflows #6514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Jun 13, 2025
Merged

Introduce evaluators for agentic workflows #6514

merged 15 commits into from
Jun 13, 2025

Conversation

shyamnamboodiripad
Copy link
Contributor

@shyamnamboodiripad shyamnamboodiripad commented Jun 11, 2025 •
edited
Loading

Introduces the following new evaluators as part of the Quality package: ToolCallAccuracyEvaluator, TaskAdherenceEvaluator, IntentResolutionEvalutor. Since the package is already GA, the new evaluators are currently marked [Experimental].

Also includes following changes:

  • Fixes a regex bug that was causing reasoning and chain of thought outputs present in the evaluation response to not be parsed correctly into the corresponding metrics.
  • Adds support for displaying tool calls and tool results in the conversation displayed in the report. This fixes [AI Evaluation] Tool messages (or anything without 'Text' contents) are not rendered in reports #6370
  • Adds support for displaying JSON content both in the conversation as well as in context (along with a new settings toggle for controlling pretty printing for the displayed JSON).
  • Adds tests for the new evaluators.

Fixes #6350
Fixes #6370

Microsoft Reviewers: Open in CodeFlow

@shyamnamboodiripad shyamnamboodiripad requested a review from a team as a code owner June 11, 2025 08:01
@github-actions github-actions bot added the area-ai-eval Microsoft.Extensions.AI.Evaluation and related label Jun 11, 2025
peterwald
peterwald approved these changes Jun 11, 2025
View reviewed changes
Copy link
Member

@peterwald peterwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@shyamnamboodiripad shyamnamboodiripad enabled auto-merge (squash) June 13, 2025 07:40
@shyamnamboodiripad shyamnamboodiripad merged commit 694b95e into dotnet:main Jun 13, 2025
6 checks passed
@shyamnamboodiripad shyamnamboodiripad deleted the agentevals branch June 16, 2025 20:16
This was referenced Jul 21, 2025
Open
Open
Open
Closed
Closed
Closed
This was referenced Jul 25, 2025
Open
Closed
Open
Open
Merged
Merged
Closed
Open
Open
Closed
Closed
Closed
Closed
Closed
Closed
Open
Merged
Open
Open
Closed
Merged
Open
Open
Open
Open
Merged
Open
Merged
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peterwald peterwald peterwald approved these changes

Assignees

@shyamnamboodiripad shyamnamboodiripad

Labels
area-ai-eval Microsoft.Extensions.AI.Evaluation and related
Projects
None yet
Milestone
No milestone
2 participants
@shyamnamboodiripad @peterwald