Skip to content

Add telemetry for tool seach tool use #3289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bhavyaus merged 1 commit into from
Jan 29, 2026
Merged

Add telemetry for tool seach tool use #3289

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions src/platform/endpoint/node/messagesApi.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ export class AnthropicMessagesProcessor {
private readonly requestId: string,
private readonly ghRequestId: string,
@ILogService private readonly logService: ILogService,
@ITelemetryService private readonly telemetryService: ITelemetryService,
) { }

/**
Expand Down Expand Up @@ -491,6 +492,24 @@ export class AnthropicMessagesProcessor {

this.logService.trace(`[messagesAPI] Tool search discovered ${toolNames.length} tools: ${toolNames.join(', ')}`);

/* __GDPR__
"toolSearchToolInvoked" : {
"owner": "bhavyaus",
"comment": "Details about invocation of tools",
"requestId": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The request ID for correlation" },
"interactionId": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The interaction ID for correlation" },
"validateOutcome": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The outcome of the tool input validation. valid, invalid and unknown" },
"invokeOutcome": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The outcome of the tool invocation. success, error" },
"toolName": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The name of the tool being invoked." },
"model": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model that invoked the tool" },
"discoveredToolCount": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Number of tools discovered", "isMeasurement": true }
}
*/
this.telemetryService.sendMSFTTelemetryEvent('toolSearchToolInvoked',
{ requestId: this.requestId, interactionId: this.requestId, validateOutcome: 'unknown', invokeOutcome: 'success', toolName: TOOL_SEARCH_TOOL_NAME, model: this.model },
Copy link

Copilot AI Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interactionId is set to the same value as requestId. In the codebase, interactionId is typically used to group multiple requests into a logical user interaction (see IInteractionService in src/platform/chat/common/interactionService.ts), while requestId identifies a single request. Using the same value for both reduces the ability to correlate multiple tool search invocations that occur within the same user interaction. Consider passing the interaction ID from the calling context if available, or document that this is intentional due to architectural constraints.

Suggested change
{ requestId: this.requestId, interactionId: this.requestId, validateOutcome: 'unknown', invokeOutcome: 'success', toolName: TOOL_SEARCH_TOOL_NAME, model: this.model },
{ requestId: this.requestId, validateOutcome: 'unknown', invokeOutcome: 'success', toolName: TOOL_SEARCH_TOOL_NAME, model: this.model },

Copilot uses AI. Check for mistakes.
{ discoveredToolCount: toolNames.length }
);

// Get the original server tool call to pair with this result
const serverToolCall = this.completedServerToolCalls.get(toolSearchResult.tool_use_id);
this.completedServerToolCalls.delete(toolSearchResult.tool_use_id);
Expand Down Expand Up @@ -519,6 +538,25 @@ export class AnthropicMessagesProcessor {
} else if (toolSearchResult.content.type === 'tool_search_tool_result_error') {
this.logService.warn(`[messagesAPI] Tool search error: ${toolSearchResult.content.error_code}`);

/* __GDPR__
"toolSearchToolInvoked" : {
"owner": "bhavyaus",
"comment": "Details about invocation of tools",
"requestId": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The request ID for correlation" },
"interactionId": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The interaction ID for correlation" },
"validateOutcome": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The outcome of the tool input validation. valid, invalid and unknown" },
"invokeOutcome": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The outcome of the tool invocation. success, error" },
"toolName": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The name of the tool being invoked." },
"model": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model that invoked the tool" },
"errorCode": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Error code if failed" },
"discoveredToolCount": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Number of tools discovered", "isMeasurement": true }
}
*/
this.telemetryService.sendMSFTTelemetryEvent('toolSearchToolInvoked',
{ requestId: this.requestId, interactionId: this.requestId, validateOutcome: 'unknown', invokeOutcome: 'error', toolName: TOOL_SEARCH_TOOL_NAME, model: this.model, errorCode: toolSearchResult.content.error_code },
{ discoveredToolCount: 0 }
);

// Get the original server tool call to pair with this error result
const serverToolCall = this.completedServerToolCalls.get(toolSearchResult.tool_use_id);
this.completedServerToolCalls.delete(toolSearchResult.tool_use_id);
Expand Down
Loading