Skip to content

feat: shuffle test suite order in run command #5051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: 3.x
Choose a base branch
from

Conversation

NivYarmus
Copy link

@NivYarmus NivYarmus commented Jul 24, 2025

Motivation/Description of the PR

Hello there.
Added an option to the run command that shuffles the order in which test suites are run.
Helpful if you want to make sure that your tests are not passing just because of a lucky side effect of a previous test.

Applicable helpers:

  • Playwright
  • Puppeteer
  • WebDriver
  • REST
  • FileHelper
  • Appium
  • TestCafe

Applicable plugins:

  • allure
  • autoDelay
  • autoLogin
  • customLocator
  • pauseOnFail
  • coverage
  • retryFailedStep
  • screenshotOnFail
  • selenoid
  • stepByStepReport
  • stepTimeout
  • wdio
  • subtitles

Type of change

  • 🔥 Breaking changes
  • 🚀 New functionality
  • 🐛 Bug fix
  • 🧹 Chore
  • 📋 Documentation changes/updates
  • ♨️ Hot fix
  • 🔨 Markdown files fix - not related to source code
  • 💅 Polish code

Checklist:

  • Tests have been added
  • Documentation has been added (Run npm run docs)
  • Lint checking (Run npm run lint)
  • Local tests are passed (Run npm test)

@kobenguyent kobenguyent requested a review from Copilot July 25, 2025 07:35
Copilot

This comment was marked as outdated.

@kobenguyent
Copy link
Collaborator

I guess that'll be helpful to update docs as well -> https://github.com/codeceptjs/CodeceptJS/blob/3.x/docs/commands.md

@NivYarmus
Copy link
Author

Added doc😁

@kobenguyent kobenguyent requested a review from Copilot August 2, 2025 05:19
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a --shuffle option to the run command that randomizes the order of test suite execution, helping to identify tests that depend on side effects from previous tests.

  • Added lodash.shuffle dependency for randomizing test file order
  • Implemented shuffle logic in the Codecept class when the shuffle option is enabled
  • Updated documentation with usage example for the new shuffle option

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.

File Description
package.json Added lodash.shuffle dependency for array randomization
lib/codecept.js Implemented test file shuffling logic when shuffle option is enabled
docs/commands.md Added documentation example for the new --shuffle option
Comments suppressed due to low confidence (1)

lib/codecept.js:185

  • The shuffle functionality lacks test coverage. Consider adding tests to verify that when the shuffle option is enabled, the test files are randomized while maintaining the same set of files.
    if (this.opts.shuffle) {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants