What AI changes about testing a website, and what it doesn't

"As a Magento developer for more than ten years, I've had to test the same journeys hundreds of times, on many different sites. 'Check that the product gets added to the cart.' 'Check that the customer account is created.' 'Check that the discount code is applied correctly.' Those are hours that weren't spent adding features, but finding bugs and regressions."

That is Arnaud, who created Qaviar. Magento, also known as Adobe Commerce, is a widely used platform for building online stores. None of the three checks he lists is hard. What makes them expensive is that they never stop coming back.

Every change to a site, however small, raises the same question: did something move that shouldn't have? To answer it, someone opens the store, adds a product to the cart, creates an account, types a discount code, and confirms that everything still behaves as it did last time. Then the next change ships, and they do it again. Same pages, same steps, same expected result.

What repeats can be handed to a machine

These checks are not optional. Without them, it is the site's customers who discover the problems, and they bring their dissatisfaction with them. Developers call this regression testing: confirming that a change in one place hasn't broken something somewhere else. It protects the business.

But the time it takes produces no value. A regression test that passes adds no feature and improves nothing. It tells you that today's site still does what yesterday's site did.

And it can be automated. A check that is performed exactly the same way every time, on the same page with the same steps, waiting for the same result, is precisely the kind of task a machine can repeat. With AI now available, we find it inconceivable that a person should still spend hours replaying the same tests by hand to see whether anything has changed.

This is the work we built Qaviar to take over. You write a test in plain language, so a project manager or a client can write and read it without being a developer. The test is then replayed identically, release after release. When a replay shows a confirmed difference, its page explains the likely cause. For us, that explanation is where the real value lies.

Automating this part does not mean automating everything, though. Some checks look like the three above and are nothing like them.

What stays human: deciding what matters

Some tests cannot be fully automated: when the case is unusual, or when the answer depends on someone's judgment. These are the riskier parts of testing, and often the ones that decide whether a release goes well. They involve particular customer profiles and exceptional situations, the ones a standard test never meets.

Take an online store that sells both to the public and to businesses. A business customer has prices negotiated with the store. At checkout, they enter a discount code. This is a case that happens every day.

Should the code apply on top of the negotiated price, replace it, or be refused? Usually nobody has written that combination into a test, because nobody planned for it. A machine can confirm that a price appears on the screen. It cannot tell you whether it is the right price. Only someone who knows the commercial rule agreed with that customer can say.

That kind of check depends on knowing what matters to the business. Who decides depends on the context. Three people can be involved.

The client is where the request starts. They don't always know what it implies in the code, and they don't need to: what counts for them is the result. Simple tests let them say in advance what they expect. "A business customer who enters a code pays this, not that" is a test a client can write.

The project manager, when they get involved, works closely with both the client and the developers. Their role is to turn what the client asks for into functional tests, the checks that say what the site must do from the user's side.

The developer always has to test their own work: check that what they built works as requested, and that they haven't introduced a bug along the way.

Which of the three makes the call varies from one project to the next. What doesn't vary is that it is a person. Automating the repetitive checks doesn't take people out of testing. It gives them back the time to think about the cases where their judgment is the test.

Drawing the line on your own site

You don't need a particular tool to make the split. Take each check your team runs before a release and ask two questions.

Is it redone identically every time? Adding a product to the cart, creating a customer account, applying a standard discount code: same steps, same expected result, release after release. Those can be handed over to automation.

Does it require thinking about a specific user or situation? The business customer with negotiated prices who types a discount code is the typical case. Those checks stay with your team. And for each one, you can usually tell who should formulate it: the client, by describing the result they expect; the project manager, by translating that into a test; the developer, for the work they have just delivered.

The same feature can sit on both sides of the line. A standard discount code on a standard order is repetitive. The same code entered by a business customer is not.

Tomorrow, open the list of checks from your last release. If there isn't one, write it down from memory. Draw a line through every check that was done exactly as it was the time before. Everything you crossed out is the part nobody on your team should be doing by hand anymore.