Add initial support for tracking token usage #15378
Conversation
There was a problem hiding this comment.
Very nice! Tested with OpenAI and Anthropic models.
Tracking Request IDs
I think it might be worth including the request id in addition to the model id, when tracking the token usage. This would allow us to show the cost of specific requests later on.
Token Usage View
The Token Usage view uses the active border color, even if it isn't active or has focus. I suggest to use a normal border color instead. As the table is not really getting focus, it should probably never use the focus colors.
Should we use date-fns to show the date relative to now (e.g. 1 week ago) and the raw date only on hover? This would feel more natural than showing the raw date. See for instance the recent chat session.
| * Gets the current token usage data for all models | ||
| */ | ||
| async getTokenUsageData(): Promise<ModelTokenUsageData[]> { | ||
|
|
| * @param model The identifier of the model that was used | ||
| * @returns A promise that resolves when the token usage has been recorded | ||
| */ | ||
| recordTokenUsage(inputTokens: number, outputTokens: number, model: string): Promise<void>; |
There was a problem hiding this comment.
Nitpick: To me, it seems more logical to put the model id as a first parameter and the second parameter is an object with input and output tokens. This way we can more easily extend what we track besides input and output tokens (thinking in the direction of image and audio).
| this.client = client; | ||
| } | ||
|
|
||
| // Store token usage records |
There was a problem hiding this comment.
| // Store token usage records |
| static readonly ID = 'ai-token-usage-configuration-container-widget'; | ||
| static readonly LABEL = nls.localize('theia/ai/tokenUsage/label', 'Token Usage'); | ||
|
|
||
| // Data will be fetched from the service |
There was a problem hiding this comment.
| // Data will be fetched from the service |
| this.title.label = AITokenUsageConfigurationWidget.LABEL; | ||
| this.title.closable = false; | ||
|
|
||
| // Load initial data |
There was a problem hiding this comment.
| // Load initial data |
| // Load initial data | ||
| this.refreshData(); | ||
|
|
||
| // Subscribe to token usage updates |
There was a problem hiding this comment.
| // Subscribe to token usage updates |
|
|
||
| .token-usage-configuration-title { | ||
| margin: 0 0 16px 0; | ||
| border-bottom: 1px solid var(--theia-panelTitle-activeBorder); |
There was a problem hiding this comment.
I think we shouldn't use the active border here. This color is somewhat reserved for components that have focus and are active. Here we use this color all the time.
|
|
||
| .token-usage-table-container { | ||
| overflow-x: auto; | ||
| border: 1px solid var(--theia-panelTitle-activeBorder); |
There was a problem hiding this comment.
I think we shouldn't use the active border here. This color is somewhat reserved for components that have focus and are active. Here we use this color all the time.
| text-align: left; | ||
| padding: 10px; | ||
| background-color: var(--theia-editor-background); | ||
| border-bottom: 1px solid var(--theia-panelTitle-activeBorder); |
There was a problem hiding this comment.
I think we shouldn't use the active border here. This color is somewhat reserved for components that have focus and are active. Here we use this color all the time.
planger
left a comment
planger
left a comment
There was a problem hiding this comment.
Very nice! Thanks for the revision. Looks good to me!
github-project-automation
bot
moved this from Waiting on reviewers
to Needs merge
in PR Backlog
eneufeld
force-pushed
the
feat/token-usage
branch
from
ab5eff3 to
aa24712
Compare
What it does
This adds logic to anthropic, openai and google providers to report token usage.
Furthermore it adds a new tab to the AiConfiguration view which shows the token usage per model.
Currently the usage is not persisted over application restart.
How to test
Open the AiConfigurationView and trigger some llm request.
Follow-ups
Breaking changes
Attribution
Review checklist
Reminder for reviewers