An Android app released in India will run on thousands of device models from dozens of manufacturers, across several Android versions, screen sizes, chipsets and memory configurations. iOS is narrower but still spans several iPhone generations and iOS versions. Testing on all of them is impossible. Testing on whatever phones the team happens to own is a lottery. The useful middle ground is a device matrix: a deliberately chosen, regularly refreshed set of configurations that represents your users and your risks.
Start from your own users, not market share
National market share figures are a reasonable starting point for a brand-new app, but once an app is live, its own analytics are far better. A premium wealth-management app and a regional-language news app have very different user bases. Firebase, the Play Console, App Store Connect and most analytics SDKs report sessions by device model, OS version, screen resolution and sometimes memory class.
Sort device-and-OS combinations by share of sessions (or of revenue, if that matters more) and look at the cumulative total. A typical pattern is a long tail: the top 10 configurations might cover around half of sessions, the top 30 perhaps three-quarters, and the rest scattered across hundreds of models with tiny shares each. You cannot chase the tail model by model. You cover it by covering the characteristics it shares with the head: OS versions, screen sizes, memory tiers and manufacturer skins.
Dimensions that actually change behaviour
- OS version. Permissions, background execution limits, notification behaviour and storage access change between Android and iOS releases. Your minimum supported version and the newest release both deserve coverage.
- Manufacturer customisation. On Android, skins from Xiaomi, Samsung, Oppo, Vivo, Realme and others change battery management, notification handling and permission dialogs. Aggressive background-app killing on some skins is a frequent cause of “notifications not arriving” complaints, and it does not appear on a stock Android emulator.
- Memory and performance tier. A large share of Indian users are on budget phones with limited RAM. Apps that work smoothly on a flagship can be killed in the background, lose form data or stutter on a low-end device.
- Screen size and density. Small screens expose truncated text and overlapping buttons, especially with larger system font settings or longer regional-language strings. Foldables and tablets matter if your users have them.
- Hardware features you use. Camera for KYC document capture, NFC, biometric sensors, GPS. If the app depends on it, the matrix needs real hardware for it.
Tiering the matrix
Not every device needs every test. A practical structure has three tiers:
| Tier | What it contains | What runs on it |
|---|---|---|
| Tier 1 (about 4 to 6 devices) | The highest-usage configurations, plus one low-end Android and the newest iOS | Full regression, every release |
| Tier 2 (about 10 to 20 devices) | Other popular models, older supported OS versions, key manufacturer skins, small screens | Smoke and critical journeys, every release |
| Tier 3 (broad) | Long-tail devices, usually via a cloud device farm | Install, launch and a few key flows, periodically or for major releases |
Revisit the matrix every quarter, and whenever a major OS version ships. Usage shifts, new budget models become popular quickly, and devices that once mattered drop off.
Emulators, simulators and real devices
The Android emulator runs a full Android system image on your computer. The iOS Simulator is different: it runs your app compiled for the Mac’s own architecture against simulated iOS frameworks, rather than emulating iPhone hardware. Both are fast to start, free, easy to reset and simple to run in parallel in CI. They are excellent for functional testing of layouts and logic, for developer feedback on every commit, and for testing across many OS versions and screen sizes cheaply.
They are poor at anything that depends on real hardware or vendor software:
- Real performance, memory pressure, thermal throttling and battery drain.
- Manufacturer skins and their background restrictions.
- Camera quality, biometrics, NFC and sensor behaviour.
- Real cellular networks, SIM-dependent flows such as SMS OTP auto-read, and handovers between Wi-Fi and mobile data.
- Touch behaviour and gestures as users actually perform them.
A sensible split is emulators and simulators for the bulk of automated functional checks early in the pipeline, and real devices for release regression, performance, hardware-dependent features and anything involving vendor behaviour. A bug found only on real devices is often exactly the bug that would have generated one-star reviews.
Own devices or rent them?
Cloud device farms (BrowserStack, Sauce Labs, AWS Device Farm, Firebase Test Lab and others) offer hundreds of real devices on demand. They are good for breadth and for Tier 3 coverage without buying and maintaining hardware. Their drawbacks are latency for interactive debugging, shared devices that may be busy at peak hours, and cost that grows with parallel usage.
An in-house lab of a dozen or so Tier 1 and Tier 2 devices is often still worthwhile for daily testing, for SIM-based flows, and for manual and exploratory sessions where a tester needs to hold the phone. The common pattern is a small owned lab plus a cloud subscription for breadth.
Signs the matrix is wrong
Production crash reports are the matrix’s report card. If crashes cluster on device models or OS versions you never test, the matrix needs updating. If most user complaints come from low-memory devices and your Tier 1 is all flagships, the matrix reflects the team’s phones rather than the customers’. Feeding crash and review data back into the matrix every quarter is what keeps it honest.
Written by the RP Strativa Team. Tier sizes and coverage proportions in the examples are illustrative.
Leave a Reply