Where AI actually fails at test migration

AI does not fail at test migration uniformly. It fails in specific places, for specific reasons. The tools that promise automated Selenium-to-Playwright conversion tell you about the places where they succeed; this article covers the rest.

The claim under examination

Several tools now market themselves as automated migration engines, with “weeks to minutes” the recurring phrase. Some of them produce useful output. The question is not whether AI assists with the mechanical parts of a migration; it demonstrably does. The question is where that assistance ends and where human judgement begins, and nobody selling the tooling is incentivised to draw that line clearly. Which is why it tends not to get drawn.

The one measurement worth trusting

The best independent data point we have comes from a TTC Global controlled study that measured GitHub Copilot with Playwright MCP against a real enterprise suite (Workday HRIS, not a sample project assembled to flatter the result). Average time saving: 24.9%, ranging from 12.8% to 36.2% depending on work type.

Both halves of that number deserve attention. A quarter of migration effort handed back to the team is a real gain, and pretending otherwise would be silly. And it is nowhere close to “minutes”. The gap between saving a quarter of the effort and automating the whole job is exactly where your suite will run into trouble if nobody is watching closely.

Where AI reliably falls short

The six categories below are not edge cases in a complex enterprise suite. They are its centre. Each one fails for a structural reason, not because the model is inattentive.

Custom wait utilities

A mature Selenium suite accumulates bespoke wait helpers such as waitForModalDismiss, waitForBatchJobComplete, waitForBalancesToReconcile. A conversion tool sees a function call and maps it to a Playwright wait construct. What it cannot see is the race condition the helper was written to address, or the two earlier implementations that failed before this one held. That timing knowledge is not in the source code; it is in the commit history and the heads of whoever wrote it.

Domain helper classes

LoginHelper, CheckoutFlow, InvoiceWorkflow. These look like automation utilities, but they are business processes expressed in test code. Converting them literally into Playwright equivalents preserves the method names and loses the intent. When a page changes, a mechanically translated CheckoutFlow will fail in ways that are harder to diagnose, because the abstraction is now in the wrong place for the new framework.

Page Object hierarchy decisions

Selenium suites often arrive with deep inheritance trees: BasePage extended by AuthenticatedPage extended by DashboardPage. That structure fitted the WebDriver programming model. Playwright’s fixture system is designed around composition, not inheritance. A faithful conversion of the hierarchy is not merely mechanical work. It is frequently the wrong answer. The correct migration may require dismantling the structure entirely, which requires understanding why it was built that way in the first place.

Setup and teardown orchestration

Shared state between tests is the most common source of subtle failures after a migration. A @BeforeClass that seeds a database assumes a particular execution order. A teardown that only runs on a passing test leaves state for the next test to find. These patterns are often invisible until you run the migrated suite in parallel. Then failures appear with no obvious cause, because the converter had no way to reason about ordering dependencies.

Complex authentication state

SSO flows, MFA challenges, and token refresh mid-run are handled in Playwright’s storageState mechanism, and that mechanism is a real improvement over anything Selenium offered. But adopting it properly is a redesign of how authentication is managed across the suite, not a translation. A tool that converts a LoginHelper into a Playwright equivalent has not taken advantage of storageState. It has moved the original problem into a slightly different shape.

JavaScript executor injection

Every call to executeScript in a Selenium suite is a place where the driver could not do what the test needed, so someone dropped into raw JavaScript to force it. Some of these are legitimate, like scrolling an element into view or triggering synthetic events. Others are workarounds for bugs in a version of Selenium from several years ago. A migration tool will convert each one faithfully. A human engineer needs to decide whether each one should still exist.

A converter translates syntax. It cannot recover judgement that nobody wrote down.

Where the line actually falls

The split is simple enough. AI handles the mechanical bulk quickly and well, and it should. Direct API mappings, import rewrites and boilerplate are exactly what a model is good at, so driver.findElement(By.id("x")) becomes page.locator('#x') without anyone needing to think about it. If your suite is predominantly straightforward locator calls and simple page interactions, the available tools will take you most of the way and a review pass will catch the rest. That is a real scenario, and those teams should use the tools directly and keep the money.

For a mature enterprise suite, one extended by multiple teams over several years, the six categories above are where most of the value and most of the risk live. The work there is not translating syntax; it is making decisions about a suite that has accumulated years of intent. That requires a human engineer in the loop as a decision-maker, not as a final auditor of AI output.

The part that compounds over time is not getting the migration done, but teaching your team to drive AI this way themselves, as a power tool with a clear account of where it stops. That capability should stay with your team after the work is finished. If you want a frank assessment of where your suite sits, write to hello@qualitylabs.eu.