A nightly run finishes at 3 a.m. IST with 23 failures out of 900. The report says, for most of them, “element not found” or “expected true but was false”. The first two hours of the next morning go into rerunning tests locally, one by one, to discover that 15 failed because a downstream API was down, 5 because of a genuine bug in the discount logic, and 3 because of a timing issue in the suite. All the information needed to know this existed at 3 a.m. It just was not captured.

The principle that fixes this is simple to state: a failed test should carry enough evidence to be triaged without being rerun. Reruns are slow, they may not reproduce intermittent issues, and in a shared environment the state that caused the failure is often gone by morning. Designing a framework around this principle changes what you log, when you capture artefacts and how the report is organised.

Start from the triage questions

Whoever opens a failure is trying to answer a short list of questions, roughly in this order:

  1. Where and on what did it run? Environment, build version, browser or device, time.
  2. Which step failed, and what was the test trying to do at that moment?
  3. What did the application look like, and what did it return?
  4. Is this a product bug, an environment problem, a test data problem or a test code problem?
  5. Has this test failed like this before?

Every piece of evidence you capture should help answer one of these. Anything else is noise that makes the useful parts harder to find.

Logs that read like the test, not like the driver

Low-level driver logs record every HTTP call between the test and the browser. They are occasionally useful, but a person cannot read them quickly. What helps more is a step log written at the level of intent: “Logging in as retail customer”, “Adding 2 units of SKU 4471 to cart”, “Applying coupon FESTIVE10”, “Expecting cart total ₹1,798, found ₹1,998”. The last line alone tells the reader that the discount was not applied, which points straight at the coupon logic.

Two habits produce logs like this. First, page objects and API clients log their own actions with meaningful parameters, so tests do not need to. Second, assertions carry messages that state the expectation and the actual value in business terms. “Expected true but was false” is the worst possible assertion message; “Order status should be CONFIRMED after payment, was PENDING” is a diagnosis.

Include identifiers that connect the test to the system: the order ID created, the user ID, and if the application exposes one, the request or correlation ID returned in a response header. With that ID, a developer can search the application’s own logs for exactly that transaction instead of guessing by timestamp.

Artefacts captured at the moment of failure

For UI tests, a screenshot at the point of failure is the minimum. It often shows the cause immediately: an error banner, a maintenance page, a cookie consent dialog covering the button. Add the page’s HTML source and the browser console log; a JavaScript error in the console frequently explains why a button did nothing.

Richer artefacts exist and are worth using selectively:

  • Network logs (HAR files or captured request and response pairs) show whether the backend returned an error, slowed down or returned unexpected data. This is often what separates “frontend bug” from “backend down”.
  • Video shows what happened before the failure, which a single screenshot cannot.
  • Traces, such as Playwright’s trace viewer, combine a timeline of actions, DOM snapshots before and after each step, network calls and console output in one file you can step through after the fact.

These are heavier, so a common compromise is to record them only on failure, or only on the first retry, which is a built-in option in Playwright. For API tests, the equivalent artefact is the full request and response for the failing call, with secrets masked.

Classifying failures automatically

Back to the 23 failures. If 15 of them share the same root cause, triage should see one problem, not fifteen. A framework can help in a few ways:

  • Run a quick health check of key dependencies before the suite, and mark the run as environment-blocked if they are down, rather than letting hundreds of tests fail noisily.
  • Group failures by exception type and message signature, so identical errors cluster together.
  • Tag failures by category where the framework can tell: an HTTP 5xx from a dependency, a timeout waiting for an element, an assertion mismatch. Reporting tools such as Allure support categories defined by message patterns.

History matters too. A test that has failed intermittently 6 times in the last 30 runs needs different treatment from one that failed for the first time today, right after a new commit. Reports that show each test’s recent pass and fail pattern make this visible at a glance.

What to leave out

Capturing everything for every test sounds safe but backfires. Screenshots after every step for 900 tests produce thousands of images, slow the suite and bury the one that matters. Logging full response bodies for every call can leak personal data into reports that many people can open, which is a concern under India’s Digital Personal Data Protection Act if test environments hold real customer data. Capture rich evidence on failure, keep passing tests lean, and mask tokens, passwords and personal fields everywhere.

The report as a front page

The report should open with the run context (environment, build, browser, time, commit), then a summary grouped by failure cause, then individual failures with their step log, assertion message and artefacts one click away. Each failure should link to its history and, where relevant, to the defect already raised for it.

A useful benchmark is the “cold reader” test. Hand a failed result to a teammate who did not write the test. If they can say what went wrong, and roughly whose problem it is, within a few minutes and without running anything, the diagnostic design is working. If their first move is to rerun it, something is missing from the evidence.

Written by the RP Strativa Team. Failure counts, SKUs and prices in the examples are illustrative.

Leave a Reply

Trending