Most broken dashboards are discovered by the wrong person. A regional manager notices that yesterday’s sales look oddly low and emails the analytics team. The team digs in and finds that an upstream job loaded only part of the data. By then the number has been in a morning review meeting and a forwarded screenshot. Data observability is the practice of making sure the data team finds out first, ideally before anyone downstream sees the problem.
Where the idea comes from
Software engineering teams have long used observability for applications: logs, metrics and traces that show whether a system is healthy and, when it is not, where the problem lies. Data observability applies the same thinking to data pipelines and tables. The term became widespread in the early 2020s, helped by vendors building tools specifically for it, but the underlying idea is simple: monitor the data itself, continuously, not just whether the jobs that move it reported success.
That last point matters. A job can finish with a green tick and still load nothing, or load yesterday’s file again, or load a file where a column has shifted by one position. Job monitoring tells you the pipe ran. Observability tells you what came out of it.
The five signals most tools watch
Descriptions vary, but most frameworks group checks into roughly five areas.
Freshness. When was this table last updated, and is that later than expected? If the orders table normally updates every hour and has not changed in four, something upstream has stalled. Freshness is the cheapest check to run and catches a large share of real incidents.
Volume. Did the expected number of rows arrive? Suppose a daily orders load has, over the past week, brought in between roughly 47,800 and 52,100 rows, averaging about 49,800 with a standard deviation near 1,500. A load of 31,000 rows sits more than 12 standard deviations below that average. Nothing failed technically, yet something is clearly wrong, perhaps one marketplace’s feed is missing. A simple statistical threshold flags it.
Schema. Did columns get added, removed, renamed or change type? An upstream team renaming cust_id to customer_id can silently turn a join into a column of nulls. Schema monitoring compares today’s structure with yesterday’s and alerts on changes.
Distribution. Do the values look normal? Null rates, distinct counts, averages, min and max, the share of each category. If the share of orders with payment mode “UPI” suddenly drops from 60% to 0% while “Unknown” jumps to 60%, a mapping broke. If average order value doubles overnight, perhaps amounts started arriving in paise instead of rupees.
Lineage. Which upstream tables feed this one, and which dashboards and models depend on it? Lineage does not detect problems by itself, but it turns an alert into an action. When the orders table fails a volume check, lineage tells you the cause is probably the marketplace ingestion job and that the affected outputs are the daily sales dashboard and the demand forecast.
Tests versus monitors
Observability is often confused with data testing, and the two complement each other.
- Tests are explicit rules written by people who know the data: order ID is unique, amount is never negative, every order has a valid store. Tools such as dbt tests or Great Expectations are typical. Tests catch known failure modes precisely.
- Monitors learn what normal looks like from history and alert on deviations: row counts, null rates, freshness patterns. They catch unknown failure modes, the things nobody thought to write a test for.
A reasonable setup uses tests for critical business rules and monitors for broad coverage across many tables, where writing individual tests would be impractical.
The alert fatigue problem
The fastest way to kill an observability programme is to monitor everything and alert on everything. Within weeks, the alerts channel has hundreds of messages, most of them harmless (volume always dips on Diwali; a test table changed schema because someone was experimenting), and people stop reading it.
Practical ways to keep signal high:
- Tier your tables. Start with the twenty or so tables that feed executive and regulatory reports. Monitor them tightly. Monitor the rest loosely or not at all.
- Account for seasonality. Volume on a Sunday or during a festival sale is not comparable to a normal Tuesday. Good anomaly detection compares like with like, such as the same weekday in previous weeks.
- Route alerts to owners. An alert that goes to a general channel belongs to nobody. Each critical table needs a named owner who receives its alerts.
- Tune after every false alarm. If an alert fired and nothing was wrong, adjust the threshold or remove the check. Treat noise as a defect.
Stopping bad data, not just reporting it
Detection is half the value. The other half comes from what happens next. Mature setups add circuit breakers: if a critical check fails, the pipeline stops before publishing to the reporting layer, so dashboards keep showing yesterday’s correct numbers rather than today’s wrong ones. Some teams also show a small status banner on key dashboards (“data delayed, last updated 6:10 AM IST”) so users know to wait rather than act on stale numbers.
This is a trade-off. Stopping the pipeline means delay; letting it run means risking wrong numbers. For a finance close report, a delay is usually far better. For an operational dashboard where roughly-right-now beats exactly-right-later, a warning banner may be the better choice.
What it does not solve
Observability tells you that data changed unexpectedly. It cannot tell you that data was wrong all along. If a field has been mis-captured at source since the system went live, its distribution looks perfectly stable, and no monitor will fire. Accuracy problems of that kind still need sampling against reality and fixes in the source process. Observability is best thought of as an early-warning system for breakage, not a certificate of correctness.
Written by the RP Strativa Team. Row counts and other figures in the examples are illustrative.
Leave a Reply