Flaky test quarantine rules that do not hide product risk

Flaky test quarantine is a temporary isolation rule for a test that fails unpredictably, but it should not become a way to hide product risk.

A good quarantine starts with classification. Is the test flaky because of timing, network dependency, shared state, filesystem order, random seed, timezone, animation, browser rendering, or real product nondeterminism? If the failure might represent a customer-visible race condition, quarantining it without an owner is dangerous. The test is noisy, but the behavior may still be real.

Set a short quarantine window. The record should include the failing test name, recent failure rate, first failing build, suspected cause, owner, and next review date. If the test blocks every deploy, it may need to be removed from the required gate temporarily. But it should still run in a separate job so failure data continues to accumulate.

Do not quarantine whole files when one assertion is unstable. Do not replace a flaky end-to-end test with no coverage unless a lower-level test already protects the same behavior. Do not merge retry logic without counting retries; retries can reduce noise while hiding a worsening failure rate.

The best fixes make the test more deterministic: control the clock, seed randomness, mock unstable network calls, wait for specific UI state rather than arbitrary time, isolate database state, or remove ordering assumptions. If the app itself is nondeterministic, fix the product path before celebrating the test fix.

Quarantine is acceptable when it buys time for diagnosis. It becomes debt when the team stops asking what the flake is teaching.

Flaky test quarantine rules that do not hide product risk

// COMMENTS

ON THIS PAGE