AI models are very good at producing code that looks right. It compiles. It is tidy. It often comes with tests that pass. And sometimes it does not do what was asked.
That last part is the problem we design around. This post explains how, in five habits.
1. The definition of done is written before the code
Every change starts as a written issue: what should happen, for whom, and how we will know. Before any code exists, a separate agent turns that issue into failing acceptance tests.
That agent sees the issue and nothing else. Not the design, not the code, not the builder’s plan. Its tests describe what was asked, in a form a machine can check.
2. The builder cannot move the goalposts
The agent that builds the change cannot edit those acceptance tests. Before merge, the system checks that they are unchanged. If one did change, the reason has to be written down and accepted in review.
This matters more with AI than with people. A person under pressure might weaken a test and feel bad about it. A model trying to get to green will weaken a test without any feeling at all, and explain the change convincingly. The rule removes the temptation from the process instead of relying on restraint.
3. Tests drive the real thing
A unit test that calls one function proves that function works in isolation. It does not prove the page loads, the background job runs, or the button does what its label says.
So anything new that runs, such as a page, a background job, a retry or a data import step, gets at least one test that drives the real entry point and checks what a user would actually see. A passing unit test alone does not count as done.
4. Features that judge are scored, not admired
Some features do more than move data. They rank, pick or write. Email-to-order in UTX Freight is one: it decides which field a value belongs to, and which of your terminals a name refers to.
Our rule for features like that: they get scored against a stated bar, on real-shaped examples, before they leave beta. Email-to-order has to pass it before it reaches customers. The test data is split. One part is used during development. The other part, the held-back set, is kept aside and used once, for the final number, so the feature cannot be tuned to the exam. Because AI graders vary from run to run, results are compared across several runs.
And the bar is written down before the work starts. If the result misses it, the report says so. We would rather ship a smaller feature that meets its bar than a bigger one with an excuse.
5. Escaped bugs get traced back
No set of tests catches everything. When a bug reaches a user, fixing it is only half the job. The other half is asking which check should have caught it.
If an existing check should have caught it and did not, the check gets fixed. If no check would have caught it, a new one is added, and it stays in the suite.
Over time this is how the suite gets better: not by writing more tests in general, but by writing the tests that real bugs prove we needed.
What this looks like from the outside
Customers never see any of this directly, and they should not have to. What they see is quieter:
- An update that does what its changelog says.
- Fewer “that used to work” phone calls after a change.
- A straight answer when they ask what changed and why.
What it costs
Writing the tests first takes time up front, and it can feel slow when the change looks easy. Splitting data and scoring a feature takes discipline when it would be faster to eyeball a few examples and call it good.
The trade is worth it. AI makes the code cheap to write. The tests are what make it safe to trust. When writing code costs almost nothing, the tests are where the real work is.
Our software factory post shows where these habits sit in the full workflow.