Four generations of self-healing tests, and the same missing person

Tests that repair themselves have been on the market for about a decade now, in four fairly distinct forms. Each generation is better engineering than the one before it. Each one has the same hole in the middle, and the hole is not technical.

First: the platform that owns your tests

The earliest of these were not really test frameworks. They were platforms. You recorded a flow in the vendor’s recorder or wrote it in the vendor’s own language, and from then on the tests lived on their servers, in their format, executed on their infrastructure. In some of them the execution behind the API was people, working through a queue.

The pitch was that you would not need test engineers. What you got instead was a description of your product that you could not read, could not run locally, could not put through code review, could not diff against the change that broke it, and could not take with you. The knowledge of how your application was supposed to behave went into an asset somebody else controlled. Healing, at this stage, meant raising a support ticket.

The structural problem is easier to see now than it was then. The vendor understood browsers extremely well. Nobody in the loop understood your business. So when a test went red because a business rule had changed, the platform had no way to tell that apart from a stale selector, and every incentive to treat it as the second one.

Second: the locator that heals itself

Then came the machine learning generation. When a selector fails, score the candidate elements on the page by attribute similarity, position, text content and neighbours, pick the closest match, carry on, and log what you did. This is genuinely clever, and the engineering behind the good implementations is real.

It is also a decision about what a red test means, made in software, on your behalf, at three in the morning.

Self-healing is a standing policy of preferring green to true.

A healing locator cannot distinguish between two situations that look identical from inside the DOM. In the first, a developer restructured a component and the button moved. Healing it is correct and saves somebody an hour. In the second, the button now reads Reserve rather than Place order, because the company moved from selling to reserving, the flow gained a deposit step, and the confirmation page no longer means what the test asserts it means. Healing that one produces a green test describing a product that no longer exists.

Both cases arrive as a failed selector. Telling them apart requires knowing what the company sells. That knowledge was never in the page.

There is a second cost, quieter than the first. A failure is information about your system, and repair-on-failure is a way of consuming that information before anyone reads it. The published data on flakiness points the same way: an empirical study across 24 projects and 10,000 suite runs found 75% of flaky tests sitting in clusters, with a mean cluster size of 13.5. Clusters of that shape come from shared fixtures, ordering dependencies and races in setup. Heal each symptom individually and the cause keeps producing new ones, for as long as you keep paying to hide them.

Third: the suite that writes itself

The current generation needs no platform and no recorder. Point a model at the application or the repository and it returns a suite. The code is fluent, idiomatic and plausible, and it arrives in volume, which is the part that impresses people in a demo.

What it does not have is any model of what the product is for. It does not know that the double-charge path is the one that ends careers, that the tax calculation covers fourteen jurisdictions with three different rounding rules, or that the failure this business genuinely fears is quiet data corruption rather than a 500. So it distributes its attention evenly across a product where risk is nothing like evenly distributed. You get breadth without priority, and a coverage figure that rises while the thing you should be afraid of stays untested. We have written separately about where this breaks down in a migration, where one controlled study measured the time saving at 24.9% rather than the minutes the category advertises.

None of that is an argument against generating tests. We generate them. It is an argument about what happens next, and in most of the suites we are shown, nothing happened next.

Fourth: the agent on the pull request

The newest form is the best of them. An agent watches the diff, writes or repairs the tests around it, and leaves a comment. It has moved to where the change actually happens, it can read the repository, and it is reviewable in the place your team already reviews things. That is real progress and we use tooling in this shape daily.

It has the diff. It does not have the reason for the diff.

It cannot see the incident three years ago that produced the guard clause it has just tidied away as redundant. It does not know that the retention period is seven years because of a letter from a regulator, and that the number is not a magic constant to be cleaned up. It does not know that an external partner depends on the exact shape of that response and has no intention of upgrading this decade. It optimises for the change in front of it, which is the one thing about a change that is already well understood, by the person who wrote it.

The interesting question about any diff is what it might break somewhere nobody is looking. Answering that needs a map of the system and its history, and that map is mostly not in the repository.

The pattern, and the part that was missing every time

Each generation moved the automation closer to the code, and each one left the same gap. That gap has never been a technology gap. These are good products, built by capable people, and the newest ones are substantially better than the oldest.

What is missing is somebody who knows what the product is for. Which failures cost money, which cost reputation, which cost nothing at all and can be left alone. Which twelve behaviours out of four hundred are the ones the business is actually built on. Every product in this history is sold, explicitly or not, as a substitute for that person, and there is no substitute for that person currently on the market.

Often there is nobody in the seat to begin with. The tool gets bought precisely because the quality function has no one able to make those calls, which means it is asked to supply the judgement as well as the labour. It can do the labour. It was never going to do the other part.

So the problem is not AI. We are not in the business of telling you to avoid it; we would be poor company for that argument, given how much of our own delivery runs on it. The problem is who is holding it, and how often the answer is nobody.

What we do instead

We use all of it. Healing suggestions, generated suites, agents on the diff. The difference is in what the output is treated as.

A healing suggestion is a triage signal, not a fix to apply automatically. Two elements matched at 80% confidence is a question worth a human answer, and the answer is frequently that the product changed and the test is now wrong about the business rather than wrong about the selector.

A generated suite is raw material. We read it, keep what asserts something worth being woken up for, rewrite what nearly does, and delete the rest, which is usually most of it. The deletions are the valuable part and no tool will make them for you, because no tool knows what your on-call engineer needs to conclude at three in the morning when a specific test goes red.

That last one is the diagnostic we keep coming back to. Take any test in your suite and ask what the person on call should conclude when it fails. If the answer is a shrug, the test is not doing anything, and healing it, generating more like it, or wiring an agent to maintain it are all ways of spending money to keep the shrug alive.

Answering that question is not a tooling problem. It is a matter of sitting with your product, your incident history and your revenue until the priorities are obvious, and then pointing the machinery at the right things. That part we do with people, and then we teach your team to do it without us. If you want somebody to look at what your current tooling has been quietly healing, hello@qualitylabs.eu is the place to start.