Getting Started

This guide walks you through integrating Unflakable with your tests, first in your local development environment and then in your continuous integration (CI) environment. Follow these steps to protect your builds from flaky tests, reduce CI costs, and boost your team's productivity.

1. Sign up for a free Unflakable account
2. Add a test suite
3. Create an API key
4. Install an Unflakable plugin
5. Run tests locally
6. Run tests in CI

Prerequisites

This guide assumes that you have existing tests that use a test framework and run in an environment supported by an Unflakable plugin. Please visit the Plugins section for more information.

Step 1: Sign up for Unflakable

If you do not already have a free Unflakable account, please sign up before continuing to the next section.

If you would like to join an existing team, please ask a member of that team to invite you. For convenience, Unflakable users can be members of multiple teams, and teams can be joined (with an invite) or left at any time.

Step 2: Add a test suite

Unflakable organizes collections of tests into test suites. We recommend adding one test suite for each set of tests that run together in a single invocation of your test framework. For example, you might have one test suite called Frontend Unit Tests and another called Frontend End-to-End Tests if these tests run in separate steps of your CI pipeline.

To add a test suite, visit the Test Suites page and click the button labeled with a plus sign (+). Choose a name for the test suite and select the appropriate test framework from the drop-down menu.

If your team is affected by flaky tests that use a test framework we don't support yet, please let us know!

Auto-quarantine

Next, decide whether you would like to auto-quarantine flaky tests. Unflakable considers a test to be flaky if it fails and then subsequently passes during retry (driven by the plugin) within a single test suite run.

If auto-quarantine is enabled, flaky tests will immediately be quarantined so that subsequent failures do not cause the test suite to fail until your team has had an opportunity to fix the test.

If you prefer, you may disable auto-quarantine and manually decide whether each flaky test should be quarantined. We strongly recommend enabling email notifications to receive notifications whenever a new test becomes flaky or quarantined.

Stable branches

Your team's workflow may involve running tests prior to merging a code change, after a merge, or both. Unflakable protects tests no matter when they run, but it distinguishes between stable and unstable code when deciding whether to quarantine tests or begin tracking new tests.

This feature depends on your team using a version control system such as Git and merging code from unstable branches into one or more stable branches, often named main or master.

In the Add Test Suite dialog, specify a regular expression that matches your team's stable branches, or use the default value. As you edit the regular expression, the dialog will automatically check the syntax and display sample matches.

If your team does not use code branches, or you only plan to use Unflakable on stable branches, select All branches are stable.

Integrations (optional)

If you have already configured one or more integrations, an optional Integrations section will appear near the bottom of the Add Test Suite dialog. Check the box next to any integrations you would like to enable for this test suite.

Click ADD

When you are finished, click ADD to create the test suite. Once the test suite has been created, instructions will appear to indicate how to run this test suite, including the test suite ID needed by the corresponding Unflakable plugin. To access these instructions again in the future, simply visit the Test Suites page and click the View Run Instructions icon ( ) in the upper-right corner of the respective test suite card.

Step 3: Create an API key

Unflakable plugins use API keys to authenticate when reporting results. To create an API key, visit the Team Settings page and click the Add API Key button labeled with a plus sign (+) in the API Keys section. Provide a description to help you identify the API key in the future, and click ADD.

Step 4: Install an Unflakable plugin

Next, install and configure the appropriate Unflakable plugin for the test suite created above. Please refer to the instructions for the appropriate plugin below.

Step 5: Run tests locally (recommended)

After following the steps above, you are ready to run your tests! We recommend running tests locally to check the configuration before committing any necessary changes to your repository and running tests in your CI environment. Follow the instructions above for running the appropriate plugin, and provide the test suite ID and API key you created earlier.

Once the tests finish running, the results are reported to Unflakable and available on the Test Suites page. Note that Unflakable only tracks new tests after they run on a stable branch. If the initial runs take place on unstable branches, you may notice that the Test Suites page doesn't list any tests.

Step 6: Run tests in CI

After running the tests locally to validate the configuration, you're ready to enable Unflakable in your CI environment!

To do so, commit any relevant configuration changes to your code repository, and configure your CI environment to set the UNFLAKABLE_API_KEY environment variable to the value of the API key created above. Please refer to your CI documentation for details on managing secrets and exposing them as environment variables. Be sure to add any command-line arguments required by the appropriate Unflakable plugin to the build scripts that run your tests.

As mentioned above, Unflakable only tracks new tests after they run on a stable branch. If the initial runs take place on unstable branches, you may notice that the Test Suites page doesn't list any tests.

Branch and commit detection

The plugin will automatically infer Git branch names and commit hashes, which are visible when viewing test results on the Test Suites page. Branch names are also used to distinguish between stable and unstable code.

If your team uses a version control system other than Git or the plugin is unable to infer the branch name and commit hash from your environment, we recommend specifying the branch name and commit hash explicitly using the environment variables or command-line arguments documented for the appropriate plugin.

Next steps

Now that your first test suite is up and running in CI and protected by Unflakable, we suggest the following next steps:

• Add integrations to receive notifications about flaky tests.
• Upgrade to a paid plan to access premium features and email support.
• Invite team members to join your Unflakable team.
• Protect additional test suites using Unflakable.
• Let us know how your experience went, and if there's anything we can do to help!