Debugging: The 9 indispensable rules for finding even the most elusive software and hardware problems. By: David J. Agans

Book review

David Agans keeps it short and practical. The book came out in 2002, and it reads like a collection of war stories from real debugging sessions. It's still relevant today, maybe even more, with cloud systems, microservices, and hardware and software mixed everywhere. I like how short it is. You can finish it in a weekend and start using it on Monday.

Summary

The book teaches you how to get from "something is wrong" to the root cause, fast. Agans insists on careful observation, disciplined experiments, and teamwork. The nine rules are timeless, and when you combine them with modern tools like dashboards, logs, traces, and feature flags, they become even more powerful.

The nine rules

1. Understand the system. Know the architecture, read the docs, know what "normal" looks like.
2. Make it fail. Reproduce the bug in a controlled way, and capture the logs, traces, and metrics.
3. Quit thinking and look. Look at the evidence before you build theories.
4. Divide and conquer. Split the code, the config, or the network path in half again and again until the problem has nowhere to hide.
5. Change one thing at a time. Isolate your variables.
6. Keep an audit trail. Write down what you tried and what happened.
7. Check the plug. Question your assumptions, even the obvious ones.
8. Get a fresh view. Ask someone else to look at it.
9. If you didn't fix it, it ain't fixed. Make sure your fix really solved the problem.

The rules look simple, but you get the most out of them when you combine them. You don't have to follow them in strict order, every bug has its own shape.

The rules in practice

Understand the system

Read the architecture docs, the onboarding wiki, and the code. Know how a request flows through your services and your hardware. In distributed systems you also need to understand the contracts between components: API schemas, message queues, retry policies, timeouts. Learn your tools and what data they can give you. And don't rely on your memory. Check the docs and the dashboards every time.

Make it fail

You can't fix what you can't see failing. Reproduce the issue step by step. Automate the failure if you can, with a failing unit test, a curl script, or a load test. Write down the exact build, configuration, and environment. For bugs that happen only sometimes, log everything, compare good runs with bad runs, and look for what's different. And don't fake the failure. Simulating the bug can hide the real trigger.

Quit thinking and look

Facts first, theories later. Open the logs, look at the database record, check the metrics. Take a screenshot of the dashboards before you restart anything. If everything looks fine, dig deeper: turn on debug logging, capture the network traffic, or attach a profiler. Evidence stops you from bending the facts to fit your favorite theory.

Divide and conquer

Cut the problem in half, again and again, until the culprit has nowhere to hide. Use git bisect when a regression appeared after some merge. In production, use feature flags or route traffic gradually to isolate the suspect component. For hardware, remove parts or swap cables and see if the problem moves. Every cut should include or exclude the bug. Keep cutting until the answer is obvious.

Change one thing at a time

Whether you are changing code, drivers, or infrastructure settings, change one thing per experiment. If you change many things at once, you won't know which one fixed it, or which one broke something else. If you must apply multiple patches, test each one separately.

Keep an audit trail

Write everything down: the time, the command, what you expected, and what happened. Keep your temporary scripts in a branch so the team can run them again. Save snapshots of the dashboards. If someone else has to pick up the problem later, they have the full context. And it makes the post mortem much easier.

Check the plug

Never skip the obvious. Check the power, the cables, the certificates, the secrets, the DNS, the permissions, and the quotas. In the cloud, expired tokens and wrong configs cause more incidents than exotic race conditions. Build health checks that fail loudly when a dependency is down.

Get a fresh view

When you run out of ideas, ask another engineer, someone from support, or even a product manager to walk through the symptoms with you. Explaining the problem out loud forces you to be clear, and half the time you find the mistake in the middle of a sentence. A fresh pair of eyes breaks your tunnel vision.

If you didn't fix it, it ain't fixed

After the fix is deployed, keep watching. Add metrics or checks that alert you if the bug comes back. Run your regression tests. Only declare victory when you see stable results over time. And if the bug disappeared without you understanding why, it's not fixed. It's hiding.

Final thoughts

Agans' nine rules are still the backbone of debugging. Combine them with today's tools, like observability platforms, CI, and remote debuggers, and you get a solid process that works for firmware, backend services, and mobile apps alike. Debugging is a craft. The more disciplined you are, the faster you become the person who fixes the "impossible" bugs.

If you want to see how I combine these rules with tests, read Debugging With TDD.