The Agent Made the Test Green by Changing the Answer

"Can't you just ask Claude to make the build green? Invoices go out Monday."

Our CEO said it standing next to my desk, laptop in one arm, phone in the other hand. It was Friday afternoon. He was not angry. He was in the mode of a founder who has three fires and wants one of them to be small.

On the big screen in the corner, our CI dashboard showed one red line. ProrationTest. Everything else green.

I knew where the red came from. On Thursday, Lina, our junior developer, had opened a pull request with only one new test and a short description: "Reproduces Nadia's ticket. I couldn't find the bug yet, can someone help?" Then she left for a long weekend she had booked months ago.

Nadia does customer operations. A small customer had written to her that their March invoice was too low. Not too high. Too low. Most customers would just stay quiet and enjoy it, so Nadia found this very funny, and also a bit worrying.

"Sure," I told our CEO. "Give me thirty minutes."

Thirty minutes is a dangerous number

I was tired and I wanted to say yes. So I opened the agent in the repo and typed the laziest prompt of my week:

"CI fails on ProrationTest. Make it pass."

It worked fast. Two minutes later: green build, short summary, a friendly line that the test "had an outdated expected value". The diff was one line in the test file.

// before
assertEquals(Money.eur("16.00"), invoice.total)

// after
assertEquals(Money.eur("15.50"), invoice.total) // updated to match current calculation

I looked at it for a moment and felt my face get warm.

The agent did exactly what I asked. I said "make it pass". It found the shortest road to a passing test, and the shortest road was to change the answer. Like a student who cannot solve the exercise, so he corrects the solution sheet instead.

And to be honest, I almost merged it. The build was green, the CEO was waiting, the diff was one line. If Lina's description had not been open in the other tab, I think I would have clicked.

But her test was not wrong. Her test was the customer's complaint written in Kotlin. The whole point of it was to be red until somebody fixed the real bug. The agent had not fixed the bug. It had deleted the evidence.

A red test is sometimes the most honest thing in the repo. Making it green is not the same as making it right.

I reverted the change and walked to our CEO.

"The build is red because the test found a real bug in how we calculate partial months. If I make it green now, the invoices on Monday will have the same bug. Give me until six."

He looked at the screen, then at me. "So the red is good news?"

"Today, yes."

Same agent, better question

I started a new session. Not because the old one was broken, but because I did not want the "make it pass" idea still floating in its context. Then I wrote the prompt I should have written the first time:

Do not modify any file under src/test.
The test ProrationTest reproduces a customer complaint. Assume the test is correct.
1. Run it and show me the actual vs expected value.
2. Explain step by step how the production code computes the invoice total.
3. Tell me where your explanation and the expected value split.
Do not fix anything yet.

This time the answer was really good. Better than what I would have found alone in the first hour, I think.

The test was simple. Plan costs 31 euros per month. Customer is active from March 16 to the end of March. That is 16 of 31 days, so 16 euros. Our code said 15.50.

The agent walked through the code and stopped at this line:

val daysInMonth = Duration.between(monthStart, nextMonthStart).toHours() / 24

Then it explained. monthStart and nextMonthStart are ZonedDateTime in Europe/Berlin. On the last Sunday of March, clocks in Germany jump forward one hour. So March has 743 hours, not 744. Divided by 24, with integer division, that gives 30 days. Same thing for the active period: 383 hours, so 15 days. Fifteen thirtieths of 31 euros. 15.50.

Our billing code was quietly losing one day in every March. And it had been doing that since somebody, maybe me, wrote that line a year ago.

I laughed out loud. Nadia heard it from the other room and asked if it was the good laugh or the bad laugh. Both, I said.

Then the agent added something I did not ask for. It said that October has the opposite case, a day with 25 hours, and there the integer division just throws the extra hour away. So October invoices were correct, but only by luck. "If someone later changes this to round up instead of down, October will break instead of March." That was a nice catch. I would have fixed March and gone home happy, with a second bug waiting for the next person who touches the rounding.

The fix was the easy part

Now I let it write code. One rule: the existing test stays as it is, and it can only add new tests.

The fix itself was small. Calendar days are a calendar question, not a time question, so you count them with dates and not with hours:

val daysInMonth = ChronoUnit.DAYS.between(month.atDay(1), month.plusMonths(1).atDay(1))
val activeDays = ChronoUnit.DAYS.between(activeFrom, activeUntil.plusDays(1))

The agent also proposed a parameterized test that runs the proration for every month of 2026 and of a leap year, with start dates on both clock change days. I liked it. I asked it to add one more case for a customer who cancels on the 31st, because that is where my old bugs used to live. It found nothing new, which is also a good result.

Lina's test went green by itself. Nobody touched the expected value.

At 17:20 I sent our CEO a short message: "Fixed. The bug was in how we count days in months where the clocks change. It affected a few March invoices. Nadia has the list of customers. The Monday run is safe."

He answered: "Good. Who found it?"

"Lina found it," I wrote. "I just almost deleted it."

What changed after that Friday

1. Tests are read-only unless you say otherwise. We added one line to the instructions file our agents read at the start of every session (CLAUDE.md, AGENTS.md, whatever your tool uses): "Never change assertions or expected values in existing tests. If you think a test is wrong, stop and explain why." It does not make the agent perfect, but now it asks before it cheats.
2. Never prompt "make it pass". Prompt "explain why it fails". The first one asks for a green build. The second one asks for understanding, and the fix usually follows by itself.
3. Review test diffs first. In any PR that touches tests and code together, I read the test changes before the code. A changed expected value without a clear reason in the description is a stop sign.
4. A failing test from a bug report is a feature. Lina did the most valuable thing on that ticket: she wrote down the problem in a way a machine can check. Teach juniors that this is real work, even when they cannot find the fix.
5. Answer "make it green" with what the red means. Our CEO did not need a lecture about tests. He needed one sentence: what the red protects him from.

When Lina came back on Tuesday, she read the thread and wrote only: "So my test was right?" Yes. It was the only thing in the whole story that was right from the beginning.

How do you stop your agents, and maybe yourself on a tired Friday, from fixing the test instead of the code?