Sit down, have your tea, and let me ask you something that sounds silly. You have a function that calculates the monthly EMI for a loan. You run it with ₹5,00,000, 10% annual interest, 24 months. It prints a number. Is the number right?

You cannot answer that by looking at the code, and you cannot answer it by running the code again. You need something outside the code that already knows the answer. Testers call that something an oracle, and the difficulty of finding a trustworthy one is called the oracle problem. Most beginners think test cases are made of inputs. They are made of inputs and an oracle, and the second half is where the thinking is.

Where expected results actually come from

Think about what you did in the EMI case. Perhaps you opened a bank’s online calculator and typed the same figures. That is one kind of oracle: a comparable product. Perhaps you worked it out in a spreadsheet using the standard formula. That is a reference implementation you built yourself. Perhaps you looked in the requirements document, which says “EMI for the sample loan is ₹23,072”. That is a specification, and it is only as good as whoever wrote it.

There are more sources than people realise. The previous release is an oracle for regression: whatever version 4.2 produced for this input, 4.3 should too, unless the change was meant to alter it. A human expert, such as the credit analyst who has done a thousand of these by hand, is an oracle. So is common sense about the domain: an EMI of ₹3 for a five lakh loan is wrong, and you do not need a formula to say so.

Each of these can be wrong in its own way. The competitor’s calculator may round differently. Your spreadsheet may assume payment at the end of the month while the product assumes the beginning. The old release may have carried a bug for three years, and now you are faithfully testing that the bug survives. An oracle is a witness, not a judge, and witnesses disagree.

When you cannot know the exact answer

Now a harder case. You are testing a search feature on a shopping site. You type “running shoes”. What is the correct list of results, in the correct order? Nobody knows. The ranking is the product’s secret sauce and there is no formula to check against.

This is where a clever idea called metamorphic testing helps. You stop asking “is this output correct?” and ask “how should the output change when I change the input in a known way?” If you add a price filter of “under ₹2,000”, the new result set must be a subset of the old one. If you search for “Running Shoes” in capitals, you should get the same results as lowercase. If you reverse the sort order, the same items should appear, just reversed. None of these tells you the results are good. All of them catch bugs.

The classic textbook example is a sine function: you cannot easily compute the sine of an arbitrary angle by hand, but you know that sin(x) and sin(180 minus x) must be equal in degrees. The relationship is the oracle.

Partial oracles and heuristics

Often you only have a partial oracle, something that can prove the output wrong but cannot prove it right. A report whose line items must add up to its grand total. A refund that must never exceed the amount paid. A date of birth that cannot be in the future. An invoice where the sum of CGST and SGST must equal the tax on the taxable value. These checks are cheap, they apply to every input you try, and they catch a surprising number of defects.

James Bach, a well-known figure in exploratory testing, collected a list of such consistency heuristics under the mnemonic HICCUPPS. It asks whether the product is consistent with its own history, with the image the company wants to project, with comparable products, with claims made in documentation, with user expectations, with itself, with its purpose, and with applicable standards or statutes. You do not need to memorise the acronym. What matters is the habit: when something looks off, ask which kind of consistency is being violated? That question turns a vague feeling into a bug report someone can act on.

The automation trap

Here is where it bites in practice. Someone writes an automated test, runs the code once, copies the printed output into the assertion, and goes home. The test now passes forever. It also proves nothing, because the oracle is the code itself. If the code was wrong that afternoon, the test enshrines the error. Some teams call these snapshot or “golden master” tests. They have a legitimate use, namely to detect change in code you do not yet understand, but they are a poor substitute for knowing what is correct.

A related trap is calculating the expected value in the test using the same logic as the product. If your test computes EMI with a copy of the production formula, both will be wrong together. Better to hard-code a few expected values taken from an independent source, even if it feels clumsy, and keep the clumsy ones small in number.

A habit worth building

Before you write the steps of any test, write one line: “I will believe this result because…” If the sentence ends with “the system said so”, stop. Find a spreadsheet, a colleague, a rule that must always hold, or a relationship between two runs. Name the oracle, and name its weakness too. A test whose oracle you cannot name is only an execution, and an execution that nobody can judge is just the program running.

Figures and names in the examples are illustrative.

Leave a Reply

Trending