Jest Plugin (JavaScript)

The Jest plugin enables users of the Jest JavaScript test framework to quarantine flaky tests and track test results.

Using this plugin involves the following steps:

1. Install the plugin
2. Configure the plugin
3. Run tests

Compatibility

This plugin runs on Linux and MacOS and maintains compatibility with the following Jest and Node.js versions:

Installation

To install the Jest plugin, add the @unflakable/jest-plugin NPM package as a development dependency:

• Yarn
• NPM

 yarn add --dev @unflakable/jest-plugin

 npm install --save-dev @unflakable/jest-plugin

Plugin configuration

The Jest plugin has two required configuration values:

1. A testSuiteId corresponding to an Unflakable test suite.
2. An Unflakable API key specified via the UNFLAKABLE_API_KEY environment variable.

Depending on your preference, specify the test suite ID and any desired configuration options in package.json or a separate file named unflakable.json (JSON), unflakable.js/unflakable.cjs (JavaScript), or unflakable.yml/unflakable.yaml (YAML). Save the file to the Jest root directory or one of its ancestors, and commit it to your code repository.

Alternatively, the test suite ID may be specified at runtime via the UNFLAKABLE_SUITE_ID environment variable. We recommend using this environment variable if you run more than one Unflakable test suite in the same repository and would like to share common configuration options between the test suites.

• package.json
• unflakable.json
• unflakable.js
• unflakable.yaml

{
  ...
  "unflakable": {
    "testSuiteId": "YOUR_TEST_SUITE_ID"
  },
  ...
}

{
  "testSuiteId": "YOUR_TEST_SUITE_ID"
}

module.exports = {
  testSuiteId: "YOUR_TEST_SUITE_ID",
}

testSuiteId: YOUR_TEST_SUITE_ID

Configuration options

In addition to the testSuiteId configuration option (or UNFLAKABLE_SUITE_ID environment variable), the Jest plugin supports the options below.

Environment variables appear in parentheses for options that may be specified in either the configuration file or the environment. If both are provided, the environment variable is used.

failureRetries — integer

gitAutoDetect — boolean

quarantineMode — enum

testSuiteId (UNFLAKABLE_SUITE_ID) — string

uploadResults (UNFLAKABLE_UPLOAD_RESULTS) — boolean

UNFLAKABLE_API_KEY (environment variable only) — string

UNFLAKABLE_BRANCH (environment variable only) — string

UNFLAKABLE_COMMIT (environment variable only) — string

Running tests

To run tests using the Jest plugin, invoke Jest as you ordinarily would, with the addition of the UNFLAKABLE_API_KEY environment variable and the following Jest command-line options:

--reporters @unflakable/jest-plugin/dist/reporter
--runner @unflakable/jest-plugin/dist/runner

For example, if package.json specifies a script labeled test that invokes Jest, use one of the commands below.

• Yarn
• NPM

UNFLAKABLE_API_KEY=YOUR_API_KEY yarn test \
  --reporters @unflakable/jest-plugin/dist/reporter \
  --runner @unflakable/jest-plugin/dist/runner

UNFLAKABLE_API_KEY=YOUR_API_KEY npm run test \
  --reporters @unflakable/jest-plugin/dist/reporter \
  --runner @unflakable/jest-plugin/dist/runner

Known limitations

This section documents edge cases that may cause unexpected behavior, along with potential workarounds.

Long test names

Jest provides hierarchical test names: the test case name passed to test() or it() prefixed with the names of any surrounding describe() blocks. Unflakable limits each name component to a maximum of 4096 bytes (using UTF-8 to encode any Unicode characters). Additionally, each name is limited to a maximum of eight components, which supports up to seven levels of nested describe() blocks.

The Jest plugin will run all tests, but Unflakable will not track or quarantine tests surrounded by more than seven describe() blocks or within any describe() block that has a name longer than 4096 bytes. In order to support use cases such as eslint.RuleTester that often yield long test names, the plugin will truncate the last name component (i.e., the one passed to test() or it()) to 4096 bytes.

As a result of this behavior, Unflakable will treat any tests that share the first 4096-byte name prefix (within the same set of describe() blocks) as a single test case. If a subset of these tests fail, Unflakable will mark the entire group of tests as flaky. If the group becomes quarantined, all of the test cases that share this prefix will be quarantined. To avoid this behavior, we recommend providing human-readable, explicit test names rather than using large test inputs as test names. For example, eslint.RuleTester accepts an optional name field for each test case that serves this purpose.

Skipping quarantined tests

If the quarantineMode configuration option is set to the non-default value skip_tests, the Jest plugin skips quarantined tests by internally leveraging Jest's --testNamePattern feature. This feature works by prepending the names of all surrounding describe() blocks to the test case name passed to test() or it(), separating each component with a space character. For example, the following test will have the combined name “a b c”:

describe("a", () => {
  describe("b", () => {
    it("c", () => {
      ...
    });
  });
});

If the test above becomes quarantined and skip_tests is used, the following test will also be skipped because its name maps to the same string “a b c”:

describe("a b", () => {
  it("c", () => {
    ...
  });
});

In addition, Jest treats the testNamePattern regular expression as case insensitive, which means that any tests whose concatenated names differ only in letter casing will be skipped if any of the matching tests is quarantined. If the tests above are skipped, tests with combined names such as “a b C” or “A B C” will also be skipped.

We recommend using skip_tests only for test suites with naming conventions that avoid the test name aliasing cases described above.