PyTest Plugin (Python)
The PyTest plugin enables users of the PyTest and
unittest Python test frameworks
to quarantine flaky tests and track test results. This plugin
leverages PyTest's out-of-the-box support
Windows is not currently supported. Please let us know if this is a priority for your team.
To install the PyTest plugin, use pip to install the
pip install pytest-unflakable
To run tests using the PyTest plugin, invoke PyTest as you ordinarily would, with the addition
UNFLAKABLE_API_KEY environment variable and the
UNFLAKABLE_API_KEY=YOUR_API_KEY pytest \ --enable-unflakable \ --test-suite-id YOUR_TEST_SUITE_ID
In addition to the required
--test-suite-id command-line arguments,
the Unflakable PyTest plugin supports the command-line arguments below:
As an alternative to the
UNFLAKABLE_API_KEY environment variable, the PyTest plugin also supports reading the API key from a file. To use this feature, omit the
UNFLAKABLE_API_KEY environment variable and provide the
--api-key-path command-line argument along with the path to the file containing the API key.
For security reasons, the API key itself cannot be passed as a command-line argument.
Name of the version control (e.g., Git) branch containing the code being tested.
When test results are uploaded to Unflakable, the branch name is compared against the test suite's stable branches setting to determine whether to update the status of each test (e.g., flaky/quarantined) based on these test results, and whether to trigger notifications. Test results from unstable branches will be visible in the Unflakable web application but will not impact test statuses or trigger notifications.
8682b9e1d521a06e6b8df4eb4f0ff6ee2273f330) or other value that uniquely identifies the commit (version control revision) of the code being tested. This value helps identify test results in the Unflakable web application.
Enables the Unflakable PyTest plugin.
By default, PyTest loads all installed plugins (i.e., packages whose names begin with
pytest-). Passing this argument instructs the Unflakable plugin to instrument the test run.
The maximum number of times to retry each failed test. To distinguish flaky tests from failing tests, the plugin retries each failed test until either it passes or this number of retries is exhausted. Note that the total number of attempts for each test is one greater than the number of retries, including the initial attempt.
Set this value to
0 to disable retries, which will also prevent any tests from automatically being marked as flaky.
Controls the behavior of quarantined tests. The following values are supported:
ignore_failures: Quarantined tests run (including retries), but failures of quarantined tests do not cause the overall test suite to fail.
no_quarantine: Quarantined tests behave like all other tests. Failures will cause the test suite to fail.
skip_tests: Quarantined tests are skipped (do not run).
This section documents functionality that is not currently supported by the Unflakable PyTest plugin.
unittest test framework provides a feature for distinguishing similar tests using
Subtests are supported in PyTest via the
pytest-subtests plugin. However, the
pytest-subtests plugin conflicts with the Unflakable PyTest plugin due to implementation details
concerning the reporting of test results.