Managing Test Suites

Unflakable organizes collections of tests into test suites. We recommend creating one test suite for each set of tests that run together in a single execution of a test framework. For example, you might have one test suite called Frontend Unit Tests and another called Frontend End-to-End Tests if these tests run in separate steps of your team's CI pipeline.

This page covers the following topics:

1. Adding a test suite
2. Running tests
3. Updating a test status
4. Updating test suite settings
5. Deleting a test suite

Adding a test suite

To add a test suite:

1. Visit the Test Suites page.
2. Click the Add Test Suite button labeled with a plus sign (+).
3. Give the test suite a name.
4. Select the appropriate test framework from the drop-down menu. If your team is affected by flaky tests that use a test framework we don't yet support, please let us know!
5. Choose whether to auto-quarantine flaky tests.
6. Select the stable branches that should trigger test status changes and notifications.

   • To treat all branches as stable, click All branches are stable.

   • To use a regular expression to specify which branches are stable, click Branches are stable only if they match the following regular expression and enter the desired regular expression.

     After each edit, the text underneath will update to show sample branches that match, or an error message if the regular expression is invalid. The regular expression must satisfy the syntax rules of the Rust regex crate. Most standard regular expression syntax is supported, but lookahead/lookbehind and back-references are not.

7. Optionally enable any existing integrations for this test suite.
8. Click ADD to create the test suite.

Once the test suite has been created, a dialog will appear with instructions for running the tests. The instructions are specific to the test framework selected above.

Running tests

The process for running tests depends on the language and test framework in use. First, add a test suite if you have not already done so.

To view instructions for running a test suite, visit the Test Suites page and click the View Run Instructions icon ( ) in the upper-right corner of the respective test suite card. For further details, refer to the appropriate plugin documentation.

Updating a test status

To quarantine a test, remove a test from quarantine, or update a flaky test to passing:

1. Visit the Test Suites page.
2. Navigate to the tests page by clicking the Tests () button for the appropriate test suite.
   • To quarantine a test, click the Quarantine Test icon ( ) near the right side of the row. Note that this icon only appears next to tests that are not currently quarantined.
   • To remove a test from quarantine and set its status to passing, click the Remove from Quarantine icon ( ) near the right side of the row. Note that this icon only appears next to tests that are currently quarantined.
   • To update a flaky test to passing, click the Update Status to Passing icon ( ) near the right side of the row. Note that this icon only appears next to flaky tests.

Updating test suite settings

To update the settings for an existing test suite, visit the Test Suites page. Refer to the instructions below for each setting.

Renaming a test suite

To rename a test suite:

1. Visit the Test Suites page.
2. Hover your mouse cursor over the name of the test suite. A rename icon ( ) will appear to the right of the test suite name.
3. Click the rename icon. The test suite name will change to a text box.
4. Enter the desired test suite name.
5. Press ENTER or click the Save icon ( ). A green checkmark will appear once the update is successful.

Changing test framework

To change the test framework associated with a test suite:

1. Visit the Test Suites page.
2. Click on the current test framework for the test suite. A drop-down menu will appear.
3. Select the desired test framework. A green checkmark will appear once the update is successful.

Toggling auto-quarantine

If auto-quarantine is enabled, the toggle switch to the right of the Auto-quarantine tests text will appear green and positioned to the right (). Otherwise, it will appear gray and positioned to the left ().

To enable or disable auto-quarantine, click the switch. A green checkmark will appear once the update is successful.

Updating stable branches

To update the stable branches for a test suite:

1. Visit the Test Suites page.
2. Hover your mouse cursor over the current stable branches value. An edit icon ( ) will appear to the left of the current value.
3. Click the edit icon. The Edit Stable Branches dialog will appear.
4. Select the desired stable branches:

   • To treat all branches as stable, click All branches are stable.

   • To use a regular expression to specify which branches are stable, click Branches are stable only if they match the following regular expression and enter the desired regular expression.

     After each edit, the text underneath will update to show sample branches that match, or an error message if the regular expression is invalid. The regular expression must satisfy the syntax rules of the Rust regex crate. Most standard regular expression syntax is supported, but lookahead/lookbehind and back-references are not.

5. Click UPDATE to confirm the change.

Toggling integrations

To enable or disable an integration for a test suite:

1. Visit the Test Suites page.
2. Expand the Integrations settings for the test suite by clicking on Integrations. If an integration is enabled, the toggle switch to the right of its name will appear green and positioned to the right (). Otherwise, it will appear gray and positioned to the left (). Note that globally disabled integrations will not appear.
3. Click a switch to enable or disable the corresponding integration for the test suite. A green checkmark will appear once the update is successful.

Deleting a test suite

To delete a test suite:

1. Visit the Test Suites page.
2. Click the Delete Test Suite icon ( ) in the upper-right corner of the respective test suite card.
3. Confirm that you want to delete the specified test suite by clicking DELETE.