Flaky tests come in clusters

Most flaky tests are not independent. They fail in groups, for a shared reason, and the repair work compounds until you account for that.

The standard response to a flaky test is locally rational. The test goes red intermittently, so you add a wait. It keeps flaking, so you wrap it in a retry. It interrupts the pipeline enough that you quarantine it and move on. None of this is careless. Each decision is pragmatic under time pressure, and the test does go quiet. Then, next sprint, a different test starts. The sprint after that, another. If this has been going on for a year and the backlog looks roughly the same size it always did, the cause is not inattention. Something structural is going on.

What the data shows

An empirical study published on arXiv in 2025 analysed 10,000 test suite runs across 24 GitHub Java projects and identified 810 flaky tests. Of those, 75% belonged to a cluster, with a mean cluster size of 13.5 tests.

That is not what most teams expect. The intuitive model is that flaky tests are independent, each with its own timing sensitivity, its own state leak, its own particular fragility. The data says that model is wrong most of the time. The majority of flaky tests fail together, in groups, for a shared reason. This is the finding that changes how you should approach the work.

What a cluster actually means

When tests fail together consistently, in correlated bursts rather than at random, it nearly always points to something they share. A common wait utility that every test calls during setup. A shared fixture that leaves state between runs. A race condition in application startup that only appears under parallel execution. One authentication helper that occasionally returns a stale token. A database seed step that sometimes does not complete before assertions fire.

The thirteen-odd tests in a mean-sized cluster are not thirteen problems. They are one problem with thirteen symptoms.

That distinction changes the unit of work entirely. If you treat each symptom as a separate failure, you run thirteen rounds of investigation, write thirteen individual patches, and add thirteen independent wait calls or retry wrappers to your suite. Nobody touches the underlying cause: the shared utility, the leaky fixture, the race in setup. It keeps running. It keeps producing new symptoms. The queue is self-refilling.

Why individual fixes do not converge

Google’s internal data, widely cited, puts flake at roughly 1.5% of all test runs, with around 16% of tests affected over time. Google has more automated testing investment than almost any organisation on earth. The numbers have not gone to zero. They have not gone to zero specifically because standard mitigation (identify, patch, quarantine, repeat) treats the surface and not the structure.

This is not a capacity problem. If roughly three-quarters of flaky tests group into clusters averaging 13.5 tests, and you fix each test individually rather than targeting the shared cause, you are doing roughly thirteen times the necessary interventions. This is a model, not a measurement, but the direction is not in doubt. Each intervention adds a retry or a wait to the codebase. Those retries and waits are themselves future maintenance. Over time, the suite slows. The retry logic becomes tangled. Anyone new to the project has no idea which waits are structural and which are patches over symptoms.

None of this reflects badly on the engineers who made those decisions. The problem is that the standard investigation workflow does not surface cluster membership. When a test goes red, you look at that test. Nothing in the normal process prompts the question: which other tests tend to fail in the same run?

Cluster first, then fix

The practical change is to group before you patch. Before assigning a flaky test to the fix queue, look at co-occurrence: which other tests tend to fail in the same run? A cluster of tests that consistently fail together warrants a single shared root-cause investigation. Find what those tests have in common, whether that is the shared utility, the fixture or the setup step, and repair at that level. One cause, one fix.

This is also where the argument for migration lands. Cluster root causes live in the shared infrastructure: wait strategy, fixtures, state setup, authentication helpers. That is the layer a migration rewrites. Moving from Selenium to Playwright replaces more than API calls. It replaces the entire test-infrastructure layer. Built-in auto-waiting, per-test browser context isolation, and granular network interception address the most common cluster causes at source rather than by wrapping symptoms.

That makes flake investigation and migration a natural pairing. The same analysis that identifies cluster causes tells you which parts of your infrastructure need replacing, and a migration gives you a clean surface to replace them on. Running the cluster analysis before you start is worth more than discovering the clusters halfway through.

One caveat. Migration does not automatically fix clustered flakiness. If the root cause is application-level non-determinism, like a background job that fires mid-test or a third-party service that occasionally times out, it travels across with you. Knowing which clusters have infrastructure causes and which have application causes is the work that determines whether migrating will actually help.

If your flaky-test backlog is roughly the same size it was twelve months ago, cluster membership is likely a large part of the reason. The analysis takes less time than another cycle of individual patches, and the results tend to change what teams decide to repair and in what order. If it would help to run it on your suite, hello@qualitylabs.eu is the right starting point.