“Can we make it real-time?” is one of the most expensive questions in analytics, because it is nearly always asked without a price tag attached. I will take a position: the default should be batch, and each step towards real-time must be justified by a decision that gets better because of it. Here is the case for that, and the case against.

Freshness is a spectrum, not a switch

Batch processing collects data and handles it in chunks on a schedule: nightly, hourly, every fifteen minutes. Streaming processes each event as it arrives, so results can be ready within seconds or less. Between them sits micro-batch, which runs small batches every minute or so and behaves almost like streaming for many purposes. Most of the “real-time dashboards” in companies are actually refreshed every few minutes, and that is usually all they need.

The useful measure is latency: the delay between something happening in the business and the number reflecting it. Ask what latency is acceptable for each use, then build only that.

The case for batch

Batch is simpler. A job reads yesterday’s data, transforms it, writes the result. If it fails, you fix it and run it again. Re-running to correct a bug or load a late file is routine. Results are consistent: every dashboard reading the 6 a.m. refresh sees the same numbers, which avoids the meeting where two people quote two slightly different “current” totals.

Batch is cheaper to operate. Compute runs for minutes and then shuts down. Streaming systems run all day and all night, need monitoring around the clock, and require specialist skills. Tools such as Apache Kafka for moving events and Apache Flink or Spark Structured Streaming for processing them are powerful, but they have their own failure modes: duplicate events, out-of-order events, state that has to be stored and recovered.

Many metrics are not stable early. Today’s revenue at 11 a.m. is incomplete, returns and cancellations arrive later, and a live figure that keeps being revised teaches people to distrust it. For finance, reconciled numbers matter more than quick ones.

The case for faster

Some decisions lose value with every minute of delay. Card fraud screening has to decide while the customer waits at the payment screen, in a fraction of a second. A ride-hailing app matching a driver to a rider, a stock-out alert on a flash sale, a power grid monitor, a delivery app re-routing riders when kitchens run late: all of these act on what is happening now. A useful test is whether someone or something takes an action on the number within the time window you are paying to shrink.

Operational monitoring also earns faster data. If checkout failures spike at 8 p.m. on a sale night, waiting until tomorrow’s batch means the sale is over. Here a five-minute refresh is typically enough, and it is far cheaper than true streaming.

A way to price the decision

  1. Name the decision and who makes it. If no one acts on it intraday, stop here and use batch.
  2. Ask how fast the world changes relative to the action. A weekly staffing roster does not benefit from per-second data.
  3. Estimate what a delay costs: lost sales, fraud losses, customer complaints. Compare that with the extra engineering and cloud cost of reducing the delay.
  4. Pick the slowest option that meets the need: daily, hourly, 15-minute, then micro-batch, then streaming.

Cost here is not only the cloud bill. It includes on-call duty, the difficulty of backfills, and the limit on how much logic you can run on an event before it is too slow.

Hybrids are normal

Real architectures mix both. One common pattern keeps a fast path for a few critical signals, such as payment failures or inventory, and a batch path for everything else, with the batch recalculating the authoritative numbers overnight and correcting whatever the fast path approximated. The price is that you now maintain two code paths that must agree, so keep the fast path small. Another pattern streams events into storage continuously but computes the reports in scheduled batches, getting reliable raw data early and clean metrics on time.

Where I would be wrong: if your business is built around instant response, such as marketplaces, payments or logistics, speed is part of the product, and investing in streaming skills early is sensible. For the typical Indian mid-size company reporting on sales, finance, HR and marketing, I would still keep nightly batch for most things, a fifteen-minute refresh for the handful of operational views, and treat anyone asking for second-level data to explain which action they will take within that second.

Leave a Reply

Trending