What Was in the Code Just a Minute Ago: What I Learned About Regression Testing in AI-Assisted Development

A test automation engineer builds two apps in his spare time—and lets AI do the coding. It works surprisingly well. So well, in fact, that testing of all things eventually gets a little neglected. A personal account of speed, trust, and what AI-assisted software development has to do with regression testing.

“Hi, I’m Lorenz Theuer. I’ve been a test automation engineer and software tester for the past seven and a half years.”

Outside my spare time, I’m also the father of a daughter who attends a parent-run kindergarten. Which means that, in my other job, I’m part of the “media team.” Along with managing email lists and turning routers off and back on again, at some point I had an idea: I’ll build an app.

Why? Whenever the kindergarten had to switch to emergency childcare, it would email the parents and then draw little pieces of paper from a vase.

True story. Also: a lot of work.

Clearly, this process needed some digital transformation.

The parent app in practice: childcare needs can be submitted separately for multiple children.

I had never developed an app before. We needed two of them. Plus a backend with kindergarten-parent-level security, because eventually this thing was supposed to become mandatory for every parent.

So I asked Chatty for a suitable tech stack: Flutter, Dart, Firebase.

How hard could it be? We had generative AI now. It could do the programming for me.

One year later, the backend was finished and both apps were live in their respective stores.

And of course I hadn’t written a single automated test.

I already do that for a living.

How Hard Could It Be?

I knew essentially nothing about Flutter, Dart, Firebase, Firestore, or Cloud Functions, so for the setup of my entire development environment, I happily allowed myself to be demoted to the AI’s automation tool.

It told me what to do. I did it.

The first few meters of the project were surprisingly relaxing.

During my first refinement sessions with Chatty, however, one thing became clear: AI couldn’t do the modeling for me.

Or, more precisely, my prompts quickly became longer and more exact. Soon they looked less like chat messages and more like actual acceptance criteria, and I had to roll back less and less often.

The domain knowledge, after all, was mine.

I couldn’t assume the AI knew that children have parents, or that parents can have more than one child.

Domain knowledge becomes software logic: even the handling of siblings has to be modeled explicitly.

Once the architecture began to emerge from that model, the results became more robust and I could afford to be less precise again. Even GUI development still worked just fine late at night.

I noticed something else, too.

I couldn’t write Dart code, but thanks to my experience with other programming languages, I could read it and understand a surprisingly large part of it immediately.

That meant the feedback loop didn’t have to wait until I tested something in the running app. My MicroMaster in Software Development and all those years of programming experience hadn’t suddenly become useless just because the AI was writing the code.

I Had Everything Under Control

So on we went: carefully sort requirements into prompts, have them built, deploy, test, adjust, happy, repeat.

The pace was intoxicating.

Two video calls with the board, feature built. Phone call with the kindergarten management, feature out. The project grew and grew in every direction.

As long as my manual tests of the latest changes passed, that was good enough for me.

There was no fixed manual regression suite. There was no automated one either.

Why, the reader may reasonably ask? Shouldn’t that, of all things, have been easy for me?

Well, the functions were highly atomic. The potential financial damage was zero. The GUI was constantly changing, so I would have been updating tests all the time.

And most importantly: I had yet to experience any major AI failure firsthand.

I had everything under control.

The day I should have started writing tests came.

And went.

It began with syntax errors from misplaced brackets and commas in the larger Dart files. Without me touching the code, they could only be eliminated by consistently moving chunks of it into separate files.

Then, suddenly, a button disappeared from the GUI.

Of course it disappeared exactly where I hadn’t looked in a while.

When I asked about it, the answer came back:

“I checked the history. The button appears to have been removed as part of the changes in commit 6f78a3c. The associated logic is no longer present in the current version. Would you like me to restore it?”

What?

Wasn’t this supposed to work the other way around?

My daughter discovers her limits every day.

I was discovering the limits of my digital programmer.

I lost time, but not faith in my testing agenda.

And so, one day, I discovered that fundamental pieces of logic had vanished from my most important Cloud Function, processLotteryResult.

The iOS app was already in the store, and my mandatory 14-day closed Android test was already underway.

In production, I had only ever tested with one child because I was focused on debugging notifications, App Attest, and Play Integrity.

This was uncomfortable.

The function that parents and the kindergarten were supposed to trust more than any other had been quietly gutted, and my beta test was just around the corner.

I had been carried away by the speed.

With limited time and the finish line in sight, I hadn’t reviewed every change as carefully as I should have and had cheerfully waved this particular gut renovation through.

The solution would have been so simple. And, in my particular case, so painfully obvious:

AI-Assisted Regression Test Automation Strategy

Huge phrase. Moderately more work.

And a remarkably effective way of reminding my digital programmer what had still been in the code a minute ago.

Clunk!

There are other things worth not forgetting over the course of a project.

Once I added a production environment, I relied on AI-generated build commands whenever I built and deployed the apps.

The fact that I now had two flags to manage in order to point builds toward dev or prod escaped me completely.

Then, suddenly, it fell out the back of my AI’s context window as well.

Clunk.

The result: my second Apple-approved Store build, complete with broken App Attest logic, was sitting there ready for anyone to download.

Naturally, I had tested it.

On my Android test device.

Which happened to be registered in Firebase with a test ID.

Since then, I’ve had scripts built for these kinds of tasks, and plenty of similar ones.

Once something has been figured out, it no longer exists only somewhere in the context of a conversation. It lives in the project.

That works surprisingly well for other things you learn along the way that aren’t obvious from the code, too.

And it saves credits.

Flowers

My builds now run through scripts. I have a second AI write my prompts and turn our refinement work into a requirements specification while it’s at it.

And the vase can go back to holding flowers.

I wrote this article in conversation, with Chatty mostly checking my work.

If you’re reading the English version, it was more the other way around.

This is where I’m at.