PyTest Plugin (Python)

The PyTest plugin enables users of the PyTestNew tab and unittestNew tab Python test frameworks to quarantine flaky tests and track test results. This plugin leverages PyTest's out-of-the-box supportNew tab for running unittest tests.

PackagePyPIpytest-unflakableNew tab
RepositoryGitHubunflakable-pythonNew tab

LinkCompatibility

This plugin maintains compatibility with the following PythonNew tab and PyTestNew tab versions:

Python Python
  • 3.7
  • 3.8
  • 3.9
  • 3.10
  • 3.11
PyTest PyTest
  • 6.2
  • 7.0
  • 7.1
  • 7.2

LinkInstallation

To install the PyTest plugin, use pipNew tab to install the pytest-unflakable package:

pip install pytest-unflakable

LinkRunning tests

To run tests using the PyTest plugin, invoke PyTest as you ordinarily would, with the addition of the UNFLAKABLE_API_KEY environment variable and the --enable-unflakable and --test-suite-id command-line arguments:

UNFLAKABLE_API_KEY=YOUR_API_KEY pytest \
  --enable-unflakable \
  --test-suite-id YOUR_TEST_SUITE_ID

Be sure to substitute YOUR_API_KEY for an Unflakable API keyNew tab and YOUR_TEST_SUITE_ID for a test suiteNew tab ID.

LinkCommand-line options

In addition to the required --enable-unflakable and --test-suite-id command-line arguments, the Unflakable PyTest plugin supports the command-line arguments below:

Link--api-key-path <string> (UNFLAKABLE_API_KEY)

Default: none

As an alternative to the UNFLAKABLE_API_KEY environment variable, the PyTest plugin also supports reading the API key from a file. To use this feature, omit the UNFLAKABLE_API_KEY environment variable and provide the --api-key-path command-line argument along with the path to the file containing the API key.

For security reasons, the API key itself cannot be passed as a command-line argument.

Link--branch <string>

Default: Auto-detected from Git (unless --no-git-auto-detect is specified)

Name of the version control (e.g., Git) branch containing the code being tested.

If specified (or auto-detected when --no-git-auto-detect is omitted), this value is compared against the test suite's stable branches setting to determine whether to update the status of each test (e.g., flaky/quarantined) based on these test results, and whether to trigger notifications. Test results from unstable branches will be visible in the Unflakable web application but will not impact test statuses or trigger notifications.

Link--commit <string>

Default: Auto-detected from Git (unless --no-git-auto-detect is specified)
Git commit hash (e.g., 8682b9e1d521a06e6b8df4eb4f0ff6ee2273f330) or other value that uniquely identifies the commit (version control revision) of the code being tested. If specified (or auto-detected when --no-git-auto-detect is omitted), this value helps identify test results in the Unflakable web application.

Link--enable-unflakable

Default: none (required)

Enables the Unflakable PyTest plugin.

By default, PyTest loads all installed plugins (i.e., packages whose names begin with pytest-). Passing this argument instructs the Unflakable plugin to instrument the test run.

Link--failure-retries <integer>

Default: 2

The maximum number of times to retry each failed test. The plugin retries each failed test until either it passes or this number of retries is exhausted. Note that the total number of attempts for each test is one greater than the number of retries, including the initial attempt.

Set this value to 0 to disable retries, which will also prevent any tests from automatically being marked as flaky.

Link--no-git-auto-detect

Default: none
Prevents the plugin from attempting to determine the current branch and commit hash from the GitNew tab repository containing the tests. If not using Git, or if auto-detection fails, consider providing this argument along with the optional but recommended --branch and --commit arguments.

Link--quarantine-mode <ignore_failures|no_quarantine|skip_tests>

Default: ignore_failures

Controls the behavior of quarantined tests. The following values are supported:

  • ignore_failures: Quarantined tests run (including retries), but failures do not cause the overall test suite to fail.
  • no_quarantine: Quarantined tests behave like all other tests. Failures will cause the test suite to fail.
  • skip_tests: Quarantined tests are skipped (do not run).

Link--test-suite-id <string>

Default: none (required)
Unflakable test suiteNew tab for which the plugin should track and report test results.

Link--unflakable-log-level <CRITICAL|ERROR|WARNING|INFO|DEBUG>

Default: none
Emits diagnostic log messages from the plugin at the specified Python logging levelNew tab.

LinkKnown limitations

This section documents functionality that is not currently supported by the Unflakable PyTest plugin.

Linkunittest subtests

The Python unittest test framework provides a feature for distinguishing similar tests using subtestsNew tab. Subtests are supported in PyTest via the pytest-subtestsNew tab plugin. However, the pytest-subtests plugin conflicts with the Unflakable PyTest plugin due to implementation details concerning the reporting of test results.