-
Notifications
You must be signed in to change notification settings - Fork 147
Improve test hygiene in ConvertSubcommandTests #934
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@swift-ci Please test |
FileManager.default.changeCurrentDirectoryPath(temporaryDirectory.path()) | ||
addTeardownBlock { | ||
FileManager.default.changeCurrentDirectoryPath(priorWorkingDirectory) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense to set the default template here as well? It seems to me like all tests except testOptionsValidation
and testTransformForStaticHostingFlagWithoutHTMLTemplate
want the default template to be set but that it's easy forget.
For example, are these tests expected to set the default template or not?
testEmitLMDBIndex
testExperimentalEnableCustomTemplatesFlag
testExperimentalEnableDeviceFrameSupportFlag
testExperimentalEnableExternalLinkSupportFlag
testExperimentalEnableOverloadedSymbolPresentation
testParameterValidationFeatureFlag
} | |
} | |
setTemplateEnvironmentVariable(testTemplateURL.path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's probably the best. I think i wasn't sure how to manage the fact that testOptionsValidation
sets the default template during the test, but i think setting it to a reasonable default and letting it mess with it later isn't a problem. I'll make that change.
It looks like the Linux CI is running Swift 5.8; was the |
this fixes an issue when tests are run sequentially; testParameterValidationFeatureFlag (added in swiftlang#897) can fail if run after testOptionsValidation, because it sets `DOCC_HTML_DIR` to a temporary directory, which is deleted after the test exits. by resetting the environment variable in a defer block, we can restore a proper environment for later tests.
this ensures that the tests aren't affected by the machine environment, for example if a gitignore'd path contains a symlink
1af9e9b
to
1d54fbe
Compare
@swift-ci Please test |
You're right 👀
This seems to have been like this for at least a month, but possibly for longer. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing these!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
} | ||
|
||
|
||
override func setUpWithError() throws { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we combine these so that we only have one set up implementation?
The order of the different set up methods are documented but it could be easier to follow the implementation if all setup was in one method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can move the existing setUp()
code into setUpWithError()
; i agree that that would make it easier to follow.
@swift-ci Please test |
Bug/issue #, if applicable: None
Summary
This PR adds some setup and teardown for the tests in
SwiftDocCUtilitiesTests.ConvertSubcommandTests
to clean up the process environment and working directory before and after each test. This resolves some test flakiness that depended on specific environmental conditions. In particular, it resolvestestParameterValidationFeatureFlag
failing when tests are run sequentially, and resolvestestTreatWarningAsError
failing when a symlink is present in the Swift-DocC directory tree.Dependencies
None
Testing
Steps:
touch test && ln -s test test2
swift test
(Important: notbin/test
, as the test only failed when run sequentially due to unclean environment)rm test test2
(to clean up your repo state)Checklist
Make sure you check off the following items. If they cannot be completed, provide a reason.
./bin/test
script and it succeeded