Write down the symptom
The first report is usually imprecise: the site is down, the form is broken, or the page is slow. I turn that into observable facts. Which URL fails? Does it fail for everyone? Did the status change after a deploy? What does the user see and what does the server return?
This short description becomes the boundary of the investigation. Without it, debugging turns into a tour of unrelated dashboards and recent code.
- Record the time and affected route
- Separate user and operator symptoms
- Note recent changes
Reduce the search space
I compare logs, metrics and recent deployments rather than trusting one source. A log can show an exception but not its frequency. A metric can show a spike but not its cause. Together they help narrow the failing boundary.
I change one variable at a time where possible. Reverting a deployment may be the right mitigation, but I still record the hypothesis it tests so the eventual fix does not become guesswork.
- Check the failing boundary
- Compare healthy and unhealthy requests
- Prefer reversible mitigations
Check boring infrastructure first
Many production failures are not deep application bugs. A missing environment variable, expired credential, wrong port, full disk or incompatible runtime can produce a confusing stack trace. I verify those basics before redesigning a service.
For containerized applications, I check the image actually running, its environment, health endpoint and dependency reachability. The code in the repository is not necessarily the code serving the request.
- Confirm the deployed version
- Check runtime and environment values
- Verify dependency health
Use a hypothesis loop
A useful debugging loop is simple: state a hypothesis, make the smallest check that could disprove it, then update the hypothesis. If the check cannot fail, it is not a useful diagnostic; it only produces activity.
This approach is slower than guessing for the first few minutes and much faster over a full incident. It also makes collaboration easier because another person can see what has already been ruled out.
- Write the expected observation
- Run a cheap discriminating check
- Record what the check changed
Turn the fix into a guardrail
Once the service is healthy, I ask what should remain behind. It might be a regression test, a deployment check, an alert, a runbook or a clearer error message. The choice depends on the failure, but there should be a concrete answer.
An incident should improve the system's ability to tell the truth. If the next person sees the same vague symptom, the team has paid for the lesson without keeping it.