Re-enable a test in SmartApplyTests.cs#382
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary
This diff adds extensive debug logging (8 WriteLog statements) to the ApplyLastSuggestionFor test helper method and adds a 60-second wait for the Apply button to appear before interacting with it.
Goal: Improve test reliability and debuggability by adding explicit waits and detailed logging to track test execution flow.
Considerations:
- The logging density is quite high. While useful for debugging flaky tests, consider whether all 8 log statements need to remain long-term, or if some could be removed once the test is stable.
- The 60-second timeout on the Apply button wait is generous but may make test failures slower to detect. Ensure this aligns with your test timeout strategy.
mkondratek
force-pushed
the
mk/fix/smart-apply-test
branch
from
5cba27a to
18b0f75
Compare
mkondratek
changed the title
| public void Print() | ||
| { | ||
| Console.WriteLine("Hello, World!"); | ||
| var mesage = "Hello, World!"; |
There was a problem hiding this comment.
The fix is to do something more obviously incorrect here. Before, sometimes Cody suggested no changes (no code snippets). Now we do an explicit typo.
Still not ideal - ideally, we have recordings and the test runs are deterministic.
|
|
||
| [VsFact(Version = VsVersion.VS2022)] | ||
| public async Task Apply_Suggestion_Is_Modifying_Point_Document() | ||
| public async Task Apply_Suggestion_Is_Modifying_Dummy_Document() |
There was a problem hiding this comment.
I wasn’t expecting this to be the fix, nor was it the outcome I aimed for, but it does solve the issue.
2 participants
The SmartApply tests were failing inconsistently due to several issues:
Problems Fixed
Apply button not clickable - The button exists in DOM with tw-hidden class on narrow viewports. Changed wait strategy from
WaitForAsync()(waits for visible) toWaitForAsync(State.Attached)(waits for element in DOM), then remove hidden class before clicking.Missing file context - File context chips weren't appearing in chat input on the first SmartApply test run, causing the LLM to respond without code suggestions (no Apply button). Added a dummy warm-up test to initialize file context functionality. The ApplyLastSuggestionFor helper now waits for context chips before sending prompts.
Insufficient timeout - Increased timeout to 60s to accommodate LLM response times.
Test file improvements - Added intentional typo to
Manager.csto ensure LLM reliably provides code suggestions.Known Issue
The dummy warm-up test is a workaround. The root cause of why file context chips don't appear on first test run needs investigation - likely a timing or initialization issue.
Test plan
Green and more deterministic CI.