Let me tell you how I would explain this to a new joiner, with a cup of tea in hand. Your checkout page is slow. Somebody messages the team channel: “is something wrong?” Think about what you would need to know, in what order, and you will have rediscovered the three pillars of observability without learning any jargon.

First question: is something wrong, and how badly?

You want a number. Requests per second, the share that fail, how long the slowest ones take. These are metrics: numeric measurements sampled over time and stored as time series. A metric does not remember individual requests. It only knows “between 10:00 and 10:01, 4,200 requests arrived and 63 returned errors”.

That forgetfulness is the strength. A counter costs the same to store whether it counts a hundred events or a hundred million, so metrics are cheap, fast to query and ideal for dashboards and alerts. Prometheus with Grafana is a very common pairing, and cloud providers have their own equivalents.

Be careful about averages. If 95 requests take 100 ms and 5 take 6 seconds, the average is about 395 ms and looks acceptable, while one user in twenty is staring at a spinner. Track percentiles (p95, p99) or histograms, which is why good metrics libraries record a distribution rather than one number.

Second question: where in the system is the time going?

The metric says checkout p99 has jumped from 400 ms to 3 seconds. But checkout calls the cart service, the pricing service, the payment gateway and the inventory database. Which one slowed down?

A trace follows one request across all those services. Each service records a span, meaning “I did this work, from this start time, for this long”, and passes a trace identifier along in the headers so spans from different machines can be stitched together. Open one slow trace and you may see the whole 3 seconds sitting in a single span: a call to the pricing service waiting on a database query. You have moved from “the system is slow” to “this hop is slow” in a minute.

Tracing has a catch. Every service must pass the identifier on, and one service that forgets breaks the chain. Storing every trace of a busy system is expensive, so most setups sample, keeping, say, one in a hundred plus any that were slow or failed. OpenTelemetry is the vendor-neutral standard for generating this data and is a sensible thing to learn.

Third question: why did this particular thing happen?

The trace shows the pricing database call is slow. Now you want details: which query, which parameters, what the error text said. That is a log: a timestamped record of a discrete event, written by your code at the moment it happened. Logs carry the specifics that numbers and spans cannot.

Two habits make logs far more useful. Write them as structured records (key-value or JSON) instead of free sentences, so you can filter on a field. And include the trace or request identifier in every line, so you can jump from a slow trace to exactly the log lines for that request. Without that link you are searching a haystack by time range and hoping.

Logs are the most expensive pillar per useful fact. They are bulky, and volume grows with traffic. Teams that log everything at debug level in production often find the logging bill and the ingestion delay become problems themselves. Never log passwords, full card numbers or Aadhaar numbers; once in a log store, personal data spreads to people and tools that should never see it, and India’s data-protection obligations apply there as well.

How they work together

Back to the slow checkout. An alert fires from a metric: error rate above 2 per cent for five minutes. You open the dashboard, see latency climbing since a 2:15 pm deployment. You pick a slow trace, find the pricing span at 2.8 seconds. You follow its trace id into the logs and read “connection pool exhausted, waiting for a free connection”. Three tools, three questions, one story. Remove any of them and you either guess or spend an afternoon grepping.

What to set up first

If you are starting from nothing, begin with metrics on the four things users feel: how much traffic, how many errors, how long it takes, and how saturated the key resources are (CPU, memory, connection pools, queue depth). Alert on symptoms, such as error rate, rather than on every cause, such as CPU at 80 per cent, which may be perfectly healthy. Add structured logs with request ids next. Add tracing when you have several services and the question “which hop?” starts costing real time; for a single application it is often premature.

Also remember that observability is not the same as monitoring. Monitoring checks for problems you predicted, such as disk full. Observability is having enough recorded detail to answer questions you did not predict, because production failures are rarely the ones in your runbook.

Figures and names in the examples are illustrative.

Leave a Reply

Trending