A pull request pipeline is only as fast as its slowest parallel job. If a suite is split across three machines and two finish in ten minutes while the third takes seventeen, every developer waits seventeen. Sharding is the practice of dividing a test suite into parts that run on separate machines at the same time, and the interesting problem is not running them in parallel but deciding which tests go where.

A suite with twelve files

Take an e-commerce UI suite organised into twelve test files. From recent CI history, their average durations in seconds are: payments 480, checkout 420, orders 360, search 300, cart 240, returns 180, coupons 150, address 100, profile 90, reviews 75, login 60, wishlist 45. The total is 2,500 seconds, a little under 42 minutes on one machine.

Spread perfectly over three machines, each would get 833 seconds. Perfect is not usually possible, because a file cannot be cut in half, but we can see how close different methods get.

Method 1: equal counts

The simplest split gives each machine the same number of files. Sorting alphabetically and taking four files each produces:

  • Machine 1: address, cart, checkout, coupons: 910 seconds.
  • Machine 2: login, orders, payments, profile: 990 seconds.
  • Machine 3: returns, reviews, search, wishlist: 600 seconds.

The pipeline takes 990 seconds, the time of the slowest machine. Machine 3 sits idle for more than six minutes. The split is fair by count and unfair by time, because test files vary enormously in length. In real suites the spread is often far wider than here.

Method 2: longest first, to the least-loaded machine

A better approach uses the durations. Sort the files from longest to shortest, then assign each one in turn to whichever machine currently has the smallest total. In scheduling theory this is the Longest Processing Time (LPT) rule, analysed by Ronald Graham in the 1960s.

Walking through it: payments (480) goes to machine 1, checkout (420) to machine 2, orders (360) to machine 3. Search (300) goes to the lightest machine, which is machine 3 at 360, bringing it to 660. Cart (240) goes to machine 2, now 660. Returns (180) goes to machine 1, now 660. And so on, always topping up the emptiest bucket. The result:

  • Machine 1: payments, returns, coupons, wishlist: 855 seconds.
  • Machine 2: checkout, cart, address, login: 820 seconds.
  • Machine 3: orders, search, profile, reviews: 825 seconds.

The pipeline now takes 855 seconds instead of 990, within 22 seconds of the theoretical best of 833, on the same hardware. The reason for sorting longest first is intuitive: big items placed early can be balanced by many small items later, whereas a big item placed last can only land on top of an already-full machine. Graham proved that LPT’s result is never worse than 4/3 minus 1/(3m) times the optimal, where m is the number of machines; for three machines that is about 22 percent over the best possible split, and in practice it usually does far better, as here.

The floor you cannot get below

Adding machines helps, but with a limit. Using the same LPT method on this suite:

MachinesIdeal (total ÷ machines)LPT result
21,250 s1,255 s
3833 s855 s
4625 s630 s
5500 s510 s
6417 s480 s
8313 s480 s

From six machines onwards the result is stuck at 480 seconds, because the payments file alone takes that long. No split can beat the longest indivisible unit. The fix at that point is not more machines but smaller units: split the payments file into several files, or shard by individual test rather than by file, if the framework and test independence allow it.

Each machine also carries fixed overhead: starting a container, installing dependencies, launching a browser, perhaps logging in. If that costs 90 seconds per machine, the four-machine pipeline takes 720 seconds rather than 630, and the cost in CI minutes grows with every machine added. Past a certain point, more shards mostly buy a bigger bill.

Keeping it working over time

Durations drift. Use a rolling average or median of recent runs, not a single run, and refresh the data regularly. Medians resist the occasional run that was slow because of a shared environment.

New tests have no history. Give them a default, such as the average duration per test, until real timings arrive.

Deterministic assignment. If shards are recalculated on every run, a test may move between machines and expose hidden order dependencies. That is useful information, but it should be investigated, not mistaken for random flakiness.

Tools already exist. Several CI services and plugins split by timing data out of the box; CircleCI’s test splitting can use stored timings, and the pytest-split plugin balances by recorded durations. Writing your own is a small exercise, as the method above shows, but check what your platform offers first.

Static shards or a shared queue

Everything above is static sharding: the split is decided before the run. The alternative is dynamic distribution, where all tests go into a shared queue and each machine pulls the next test when it becomes free. A dynamic queue balances itself automatically, even when a test runs much slower than usual or a machine starts late, and it needs no timing history. It does require coordination infrastructure, and results are spread across machines in a less predictable way, which makes debugging order-dependent failures harder.

Static, duration-based sharding is usually the right first step: easy to implement, easy to reason about, and a large improvement over equal counts. Teams move to dynamic queues when suites are very large or timings are too unstable for history to predict.

Written by the RP Strativa Team. The suite and durations are illustrative; the calculations were checked.

Leave a Reply

Trending