A new analyst joins a food delivery company and is asked for weekly active customers in Hyderabad. The warehouse has tables called customers, dim_customer, customer_master_v2, users and cust_snapshot_daily. Which one is right? Does “active” mean ordered, or opened the app? Is Hyderabad a city field or a delivery zone? Is any of this data still being updated? Without help, the analyst messages three colleagues, waits a day, and gets three slightly different answers. A data catalogue is meant to replace that chain of messages with a search box.
Metadata: the raw material
A catalogue is built from metadata, which is simply data that describes other data. It is useful to split it into three kinds, because they come from different places and are maintained by different people.
Technical metadata describes structure: database, schema, table and column names, data types, partitions, row counts, the SQL that builds a view. It can be harvested automatically from warehouses, lakes and BI tools, so it is the easiest part to get.
Business metadata describes meaning: what a table represents, what a column means in business terms, the official definition of “active customer”, who owns the data, whether it contains personal data, which glossary term it maps to. This cannot be harvested. People have to write it, which is why it is usually the thinnest part of any catalogue.
Operational metadata describes behaviour: when a table last refreshed, whether its quality checks passed, how often it is queried and by whom, which dashboards use it, lineage to its sources. This comes from pipeline logs, query history and monitoring tools.
Each kind answers a different analyst question. Technical metadata answers “what is in it?”. Business metadata answers “what does it mean, and may I use it?”. Operational metadata answers “can I trust it today, and do other people rely on it?”.
What a good catalogue entry looks like
Go back to the five customer tables. A useful catalogue would show, for dim_customer: a one-paragraph description (“one row per customer, current and historical versions, source of truth for customer attributes”), an owner (the customer data product team), a certified badge, refresh status (updated 05:40 IST today, all tests passed), the grain, the top downstream dashboards, and a glossary link explaining how “active customer” is defined and which measure computes it. For customer_master_v2 it might show a deprecated flag and “do not use, replaced by dim_customer in 2025”. For users it might show that it is the app login table, not a customer table, and that it contains personal data with restricted access.
With that, the analyst’s choice takes minutes. Notice that the most valuable pieces (description, certification, deprecation, definition) are all business metadata, written by people.
Glossary and catalogue are not the same thing
A business glossary defines terms in business language, independent of any table: “Gross Merchandise Value: total value of orders placed including taxes and delivery fees, before cancellations and refunds.” A catalogue lists data assets. The power comes from linking them: the glossary term points to the certified columns and measures that implement it, and each column points back to the term. When finance changes the GMV definition, the glossary shows exactly which assets need updating.
How catalogues help beyond search
- Access and privacy. Tagging columns that hold personal data, such as phone numbers, addresses or Aadhaar-linked fields, lets access policies and masking rules be applied by tag rather than table by table. It also makes it possible to answer “where do we store customer phone numbers?” during a privacy review.
- Reducing duplicate work. If an analyst finds that a churn feature table already exists and is maintained, they reuse it instead of building a fourth version.
- Safe clean-up. Usage metadata shows which tables nobody has queried in six months, which makes deprecation decisions evidence-based.
- Onboarding. New joiners learn the data landscape from the catalogue rather than from whichever colleague has time.
Why many catalogues become ghost towns
A familiar pattern: a company buys or deploys a catalogue tool, connects it to every system, and within a week it lists forty thousand tables with no descriptions. Analysts search once, find nothing helpful, and go back to asking colleagues. The tool is not the problem. Harvesting technical metadata is easy; the business metadata that makes a catalogue useful requires ongoing human effort, and nobody was asked to do it.
Approaches that tend to work better:
- Curate a small core first. Document and certify the fifty or so tables behind the most important reports. Let the long tail stay undocumented but clearly marked as uncertified.
- Write metadata where the work happens. Descriptions kept in the transformation code (dbt model documentation, for example) are updated in the same change as the logic and synced to the catalogue, rather than maintained separately in a web form.
- Make ownership explicit. Every certified asset has a named owner responsible for its description and quality status.
- Put it in the analyst’s path. Integrations that show descriptions and certification inside the SQL editor or BI tool get used far more than a separate portal.
Metadata as a strategic asset
Metadata management used to be seen as documentation, the thing teams did when they had spare time. That has changed for two reasons. Regulation now expects organisations to know where sensitive data lives and how reported numbers are produced. And AI assistants that write SQL or answer business questions over company data depend heavily on good metadata: a language model can only choose the right table and the right definition of “active” if those descriptions exist and are accurate. Well-maintained metadata is increasingly what separates analytics that can be automated safely from analytics that cannot.
Written by the RP Strativa Team. The company and table names in the examples are illustrative.
Leave a Reply