I ran into a latency bug in a system with dashboards, traces and a clean 200 ms SLO.
Users were still waiting 2–3 seconds.
The error rate was low. CPU and memory looked fine. The dashboards stayed green. The user experience was telling a different story.
The problem lived outside what the tools measured. A single async path blocked a thread pool under specific conditions. It happened rarely enough that it never appeared on the charts I was looking at.
The useful question was whether the existing measurements accounted for the wait users experienced.
Three numbers, three meanings
- 200 ms: the SLO described in my original account. It was reported as met; that figure is a target, not a measured duration for every part of the workflow.
- 2–3 seconds: the user waits that prompted the investigation.
- Approximately 2 seconds → 0.2 seconds: the reported worst-case latency comparison after removing the bottleneck.
The user symptom and the before-and-after comparison are separate observations. The account does not provide the exact SLO definition, the complete timing boundary, or a breakdown assigning time to individual components.
The gap in the measurement
The reported SLO was being met. The workflow still had a stall.
The diagram shows that scope gap conceptually. It is not an execution trace or a reconstruction of the original timing breakdown.
In this case, averages hid the problem and the percentiles available to me did not expose it. That makes the coverage of the measurement a diagnostic question: which paths, conditions and observations does this view actually include?
The question was bigger than whether a service response looked fast. It was whether the measurements accounted for the waiting users experienced.
Hands-on profiling and understanding how the workflow executed were necessary to get beyond that gap.
A small fix, a bounded result
The fix was one line of code.
The worst-case latency I reported dropped from approximately 2 seconds to approximately 0.2 seconds after removing the bottleneck. Those figures describe the reported worst-case comparison; the 2–3-second waits describe the earlier user symptom.
That result connects the identified bottleneck to an improvement in the reported comparison. It does not establish the latency of every workflow or identify the complete measurement procedure.
Questions to ask when the numbers and the workflow disagree
I would use these questions to structure a similar investigation:
- Where does the user's action begin and end? Name the action and the point at which the user can continue, then compare those boundaries with the existing timers.
- Which observations are included? Identify the conditions, population and time window represented by the green indicator. Keep failed, timed-out and incomplete attempts visible.
- What explains a slow example? Preserve an actual slow observation and use profiling to investigate the path involved. Treat its behavior as something to establish.
- What work happens outside those timers? Map the workflow before assuming that a healthy component describes the complete user wait.
- What will demonstrate improvement? State the observation to compare, the relevant conditions and the result that would challenge the diagnosis.
The proposed worksheet in Measure the Workflow, Not Just the Request makes those boundaries explicit. Before a Performance Fix, Write the Prediction helps define the comparison before making a change.
This retrospective expands my December 11, 2025 LinkedIn account. That concise account reports the diagnosis and outcome without the runtime, trigger conditions, code, workload, sample count or complete benchmark setup. The questions above are a proposed method for a similar investigation.