Cypress Plugin (JavaScript)

The Cypress plugin enables users of the Cypress JavaScript test framework to detect, manage, and quarantine flaky tests.

This page covers the following topics:

1. Installation
2. Running Cypress
   1. Command-line options
3. Configuration
4. Known caveats

Compatibility

This plugin maintains compatibility with the following Cypress and Node.js versions:

Installation

To install the Cypress plugin, add the @unflakable/cypress-plugin NPM package as a development dependency:

• Yarn
• NPM

 yarn add --dev @unflakable/cypress-plugin

 npm install --save-dev @unflakable/cypress-plugin

Running Cypress

To run tests with the Cypress plugin, replace the cypress run command with cypress-unflakable. This command automatically configures Cypress to use the plugin and invokes cypress run.

The plugin requires two configuration values:

1. A test suite ID corresponding to an Unflakable test suite. Be sure to add a test suite and select Cypress as the test framework before running tests using the plugin.
2. An Unflakable API key specified via the UNFLAKABLE_API_KEY environment variable.

To get started, run:

• Yarn v2+
• Yarn v1
• NPM

UNFLAKABLE_API_KEY=YOUR_API_KEY yarn exec \
  cypress-unflakable --test-suite-id YOUR_TEST_SUITE_ID

UNFLAKABLE_API_KEY=YOUR_API_KEY yarn exec -- \
  cypress-unflakable --test-suite-id YOUR_TEST_SUITE_ID

UNFLAKABLE_API_KEY=YOUR_API_KEY npm exec \
  --package=@unflakable/cypress-plugin \
  -c "cypress-unflakable --test-suite-id YOUR_TEST_SUITE_ID"

For convenience, we recommend adding a command to the scripts section of package.json. For example, use yarn test:ci or npm run test:ci (and set the UNFLAKABLE_API_KEY environment variable) to run the script configured in package.json below:

{
  ...
  "scripts": {
    "test:ci": "cypress-unflakable --test-suite-id YOUR_TEST_SUITE_ID"
  }
}

Cypress command-line options

The cypress-unflakable command supports the command-line options listed below, as well as any Cypress command-line options for cypress run. Use “--” to separate cypress-unflakable options (if any) from cypress run options. Any options after “--” will be passed directly to Cypress. For example, the following command runs Cypress component tests by passing the --component option to Cypress:

cypress-unflakable --test-suite-id YOUR_TEST_SUITE_ID -- --component

Plugin command-line options

The cypress-unflakable command supports the following command-line options:

--[no-]auto-config

--[no-]auto-support

--branch <string>

--commit <string>

--failure-retries <integer>

--[no-]git-auto-detect

--quarantine-mode <ignore_failures|no_quarantine|skip_tests>

--test-suite-id <string>

--[no-]upload-results

--help

--version

Manual registration

By default, cypress-unflakable transparently wraps the Cypress configuration file to register the plugin. It does so by passing the --config-file option to cypress run with the path to the @unflakable/cypress-plugin/config-wrapper or @unflakable/cypress-plugin/config-wrapper-sync module. This option initially replaces the user-supplied Cypress configuration (e.g., cypress.config.js). After the wrapper module loads, it also loads the user-supplied Cypress configuration. In most cases, this process is transparent and should respect all of the user-supplied configuration values, including running the setupNodeEvents function.

For advanced use cases that conflict with the automatic configuration process, the plugin can be registered manually instead. To do so, pass the --no-auto-config option to cypress-unflakable and define a setupNodeEvents function in your Cypress configuration file that calls the registerUnflakable() function and returns its return value. We also recommend using the cypress-on-fix package, which works around a known Cypress issue to keep multiple Cypress plugins from conflicting:

• CommonJS
• ESM/TypeScript

const { registerUnflakable } = require("@unflakable/cypress-plugin");
const cypressOnFix = require("cypress-on-fix");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // e2e testing node events setup code
      ...

      return registerUnflakable(on, config);
    },
  },
  component: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // component testing node events setup code
      ...

      return registerUnflakable(on, config);
    },
  },
});

import { registerUnflakable } from "@unflakable/cypress-plugin";
import cypressOnFix from "cypress-on-fix";

export default defineConfig({
  e2e: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // e2e testing node events setup code
      ...

      return registerUnflakable(on, config);
    },
  },
  component: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // component testing node events setup code
      ...

      return registerUnflakable(on, config);
    },
  },
});

Please also let us know about your use case by contacting Support or filing a GitHub issue so that we can help make the Cypress plugin work transparently for as many scenarios as possible.

Support file

When the quarantine mode is set to skip_tests (via --quarantine-mode or the quarantineMode configuration option), the plugin transparently wraps the Cypress support file to register the Mocha instrumentation needed to skip any quarantined tests. It does so by generating a temporary support file that loads both the Unflakable Mocha instrumentation and the existing user support file (if any).

For advanced use cases that conflict with this process, the Mocha instrumentation can be registered manually instead by passing the --no-auto-support option to cypress-unflakable and adding the following code to your Cypress support file:

• CommonJS
• ESM/TypeScript

const {
  registerMochaInstrumentation,
} = require("@unflakable/cypress-plugin/skip-tests");
...

registerMochaInstrumentation();

import { registerMochaInstrumentation } from "@unflakable/cypress-plugin/skip-tests";
...

registerMochaInstrumentation();

When manually registering both the plugin (see above) and the Mocha instrumentation, pass the autoSupportFile: false option to registerUnflakable in the Cypress configuration file instead of passing the --no-auto-support option to cypress-unflakable (which has no effect when --no-auto-config is passed):

• CommonJS
• ESM/TypeScript

const { registerUnflakable } = require("@unflakable/cypress-plugin");
const cypressOnFix = require("cypress-on-fix");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // e2e testing node events setup code
      ...

      return registerUnflakable(on, config, { autoSupportFile: false });
    },
  },
  component: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // component testing node events setup code
      ...

      return registerUnflakable(on, config, { autoSupportFile: false });
    },
  },
});

import { registerUnflakable } from "@unflakable/cypress-plugin";
import cypressOnFix from "cypress-on-fix";

export default defineConfig({
  e2e: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // e2e testing node events setup code
      ...

      return registerUnflakable(on, config, { autoSupportFile: false });
    },
  },
  component: {
    setupNodeEvents(baseOn, config) {
      // Work around https://github.com/cypress-io/cypress/issues/22428 to prevent
      // multiple plugins from conflicting.
      const on = cypressOnFix(baseOn);

      // component testing node events setup code
      ...

      return registerUnflakable(on, config, { autoSupportFile: false });
    },
  },
});

Please also let us know about your use case by contacting Support or filing a GitHub issue so that we can help make the Cypress plugin work transparently for as many scenarios as possible.

Configuration

Many of the plugin's command-line options can instead be stored in a configuration file. Depending on your preference, specify the test suite ID and any desired configuration options in package.json or a separate file named unflakable.json (JSON), unflakable.js/unflakable.cjs (JavaScript), or unflakable.yml/unflakable.yaml (YAML). Save the file to the directory containing your Cypress configuration file (e.g., cypress.config.js or cypress.config.ts) or one of its ancestors, and commit it to your code repository.

Alternatively, the test suite ID may be specified at runtime via the UNFLAKABLE_SUITE_ID environment variable or --test-suite-id command-line option. We recommend using the command-line option or environment variable if you run more than one Unflakable test suite in the same repository and would like to share common configuration options between the test suites.

• package.json
• unflakable.json
• unflakable.js
• unflakable.yaml

{
  ...
  "unflakable": {
    "testSuiteId": "YOUR_TEST_SUITE_ID"
  },
  ...
}

{
  "testSuiteId": "YOUR_TEST_SUITE_ID"
}

module.exports = {
  testSuiteId: "YOUR_TEST_SUITE_ID",
}

testSuiteId: YOUR_TEST_SUITE_ID

Configuration options

The Cypress plugin supports the configuration options below. Environment variables appear in parentheses for options that may be specified either as command-line options, in the configuration file, or as environment variables. Command-line options take precedence over environment variables, which take precedence over configuration files.

failureRetries — integer

gitAutoDetect — boolean

quarantineMode — enum

testSuiteId (UNFLAKABLE_SUITE_ID) — string

uploadResults (UNFLAKABLE_UPLOAD_RESULTS) — boolean

UNFLAKABLE_API_KEY (environment variable only) — string

UNFLAKABLE_BRANCH (environment variable only) — string

UNFLAKABLE_COMMIT (environment variable only) — string

Known caveats

This section documents important caveats about how the Unflakable Cypress plugin interacts with Cypress.

Cypress retries

The plugin leverages Cypress's built-in support for test retries in order to distinguish flaky tests from failing tests (see the --failure-retries command-line option or failureRetries configuration option).

Ordinarily, Cypress treats tests that pass during a retry as passes, which hides flaky tests. For example, if a test fails twice and then passes on the third attempt, Cypress considers the test to have passed. By contrast, the Unflakable Cypress plugin treats tests that pass during retry as flaky, and cypress-unflakable will exit with a failing (non-zero) exit code if any non-quarantined tests exhibited flakiness during a Cypress run.

Mocha before/after hooks

Both Cypress and the Unflakable Cypress plugin treat Mocha beforeEach and afterEach hooks effectively as part of the test they immediately precede/follow, respectively. If either the test or one of these hooks fails, both the test and any associated beforeEach/afterEach hooks will be retried. If a test is quarantined, failures in its beforeEach/afterEach hooks will be handled as if the test itself had failed, according to the configured --quarantine-mode.

Cypress treats before and after hooks, which run only once per spec file or describe() block, differently from beforeEach/afterEach hooks. When one of these hooks fails, Cypress does not retry the hook or any of the tests that depend on it. It also skips any remaining tests that depend on a failed before hook. Because the Unflakable Cypress plugin depends on Cypress for test retries, it is unable to detect flakiness associated with before and after hooks. Instead, a failed before hook will be treated as if the first test (which will be skipped) had failed, and a failed after hook will be treated as if the last test to run had failed.

Mocha reporters

Cypress allows users to configure built-in or custom Mocha reporters. These reporters emit test results in a variety of output formats (e.g., human-friendly terminal output, HTML, etc.). The Unflakable Cypress plugin replaces the default spec reporter with a modified version that supports flaky and quarantined tests, along with improved output for retried tests.

The plugin will load any configured non-default Mocha reporters, which will continue to receive test results. However, other reporters will not distinguish between flaky and failed tests, nor will they support quarantining of failures. Consequently, other reporters may report that a test has failed, while the plugin's terminal output may show that a test was flaky and/or quarantined.

Cypress Cloud

The Unflakable Cypress plugin is currently incompatible with Cypress Cloud.