Oh My Learning

Slow Tests Economics

xunit-test-patterns/08 drafted · 8 min read

Slow Tests and Suite Execution Economics#

TL;DR#

A test suite's value is proportional to how often it's actually run, and run frequency collapses non-linearly once a suite crosses certain psychological thresholds — seconds for a "run on every save" habit, low minutes for "run before every commit," longer for "only run in CI." Slow Tests is the smell that names this specific failure mode, and the fix is almost always a deliberate, tiered suite architecture, not just "make everything faster."

The idea#

Tests only protect you if they run. A brilliant, thorough suite that takes 45 minutes to run gets run in CI once per push and essentially never locally — which means the feedback loop that makes tests valuable (write code, get told in seconds whether you broke something) never actually happens for most changes. Meszaros treats suite speed as an economic property of the whole suite, not a per-test nicety, because the cost of a slow suite isn't paid in CPU time — it's paid in the behavior change it causes: engineers stop running it, and defects that a fast suite would have caught in seconds now surface hours or days later, when they're much more expensive to diagnose and fix.

This is why "Slow Tests" gets its own smell name and its own chapter, distinct from the correctness-focused smells like Fragile or Obscure Test: a suite can be perfectly correct and well-structured and still be economically worthless if nobody runs it.

How it works#

Where slowness comes from#

The dominant source, in Meszaros's experience and in most real codebases, is unnecessary real I/O: tests that hit a real database, make real network calls, read/write real files, or sleep on real timers, when a Fake Object or Stub (see xunit-test-patterns/05) would have been fast and just as valid for the behavior actually being tested.

# slow: opens a real database connection and does real I/O for a pure calculation test
test "invoice total includes 8% tax":
    conn = openRealDatabaseConnection()   # ~150ms just to connect
    invoice = InvoiceRepository(conn).findById(42)   # another round trip

    total = invoice.calculateTotal()

    assert total == 108.00
    conn.close()

If calculateTotal()'s logic doesn't actually depend on the database at all (it's a pure calculation over already-loaded fields), the database round trips are pure overhead being paid by every single run of this test, for zero additional confidence.

# fast: builds the invoice directly, no I/O
test "invoice total includes 8% tax":
    invoice = anInvoice().withSubtotal(100.00).withTaxRate(0.08).build()

    total = invoice.calculateTotal()

    assert total == 108.00

Multiply the ~150ms saved by hundreds or thousands of similar tests, and the difference between "suite runs in 3 seconds" and "suite runs in 8 minutes" is exactly this kind of unnecessary I/O, repeated at scale.

The test pyramid as the structural answer#

Meszaros's broader answer isn't "make every individual test faster" — it's architect the suite in tiers, each with a different speed/realism trade-off: - Unit tests (the large base): pure in-memory, no I/O, Fresh Fixture with Fakes/Stubs for any DOC. Should run in milliseconds each, thousands per second in aggregate. Run on every save. - Integration/component tests (a smaller middle layer): exercise real boundaries (a real database, a real HTTP call to a test double server) deliberately, to catch the class of bug unit tests structurally cannot see (wrong SQL, wrong serialization). Run before commit/push. - End-to-end tests (a small top layer): exercise the whole real system, slow and sometimes flaky by nature. Run in CI, not on every save.

The economic argument: most bugs are logic bugs, catchable by the fast, numerous unit tier; a small number of integration-boundary bugs need the slower middle tier; and the smallest, slowest, most valuable-per-test top tier exists specifically for the risks the other two structurally can't cover (real wiring, real deployment configuration).

Worked example: diagnosing a suite that "used to be fast"#

A suite that ran in 4 seconds a year ago now takes 6 minutes. A common pattern behind this drift: every new feature added a handful of tests that reached for the existing (real) test database "because that's what the other tests near it do," rather than reaching for a Fake. No single test addition looked expensive; the aggregate did. The fix isn't punishing the newest tests — it's auditing the suite for the DOCs that are real when they don't need to be, and systematically replacing them (see xunit-test-patterns/05), then re-tiering: moving the genuinely-necessary-real-I/O tests into a slower, less-frequently-run integration tier so the unit tier stays fast.

When slowness is a signal, not just a cost#

Sometimes a stubbornly slow unit test reveals a design problem in the SUT itself, not just a test-double choice — if a "unit" genuinely cannot be tested without a real database because its logic and its persistence are entangled (no seam to inject a Fake), that's Meszaros's cue that the production code, not just the test, needs refactoring (extracting the pure logic from the I/O, a pattern often called Hexagonal/Ports-and-Adapters at the architecture level). Slow Tests can be the canary for a coupling problem worth fixing at the source.

Pros#

  • Restores the fast local feedback loop that makes tests actually change developer behavior (catch bugs in seconds, not hours).
  • Tiering the suite lets you keep genuinely valuable slow tests (true end-to-end coverage) without letting them tax every single local run.
  • Frequently surfaces real production-code coupling problems (I/O entangled with logic) as a side effect of chasing speed.

Cons#

  • Building and maintaining Fakes for every previously-real DOC is real, ongoing engineering investment (see the Fake-drift risk in xunit-test-patterns/05).
  • A multi-tier suite adds process complexity: someone has to decide what belongs in which tier, and enforce it as the codebase grows, or drift creeps back in.
  • Over-aggressively cutting integration/E2E coverage in the name of speed can leave real, boundary-level bugs (a subtly wrong SQL query, a serialization mismatch) uncaught until production.

Alternatives#

  • Parallelizing the existing slow suite — a purely infrastructural fix (more CI workers) that reduces wall-clock time without addressing the root cause; useful as a stopgap, but doesn't restore the fast local feedback loop that's the actual point.
  • Test selection / impact analysis (running only tests affected by a given change) — a sophisticated tooling-based alternative to tiering; effective at scale but adds tooling complexity and can miss non-obvious dependencies if the impact analysis is imperfect.
  • Accepting a slower suite for a small, early-stage codebase — a legitimate choice when the whole suite is still small enough (seconds) that tiering would be premature process overhead; revisit once suite time crosses the "people stop running it" threshold.

When to use it#

Treat suite speed as a first-class design concern from early on, and act the moment local runs start taking long enough that you notice yourself hesitating to run them — that hesitation is the actual economic damage this smell describes, and it happens well before a suite becomes "unbearably" slow in absolute terms.

When NOT to use it#

Don't over-invest in tiering and Fake-building for a genuinely small suite where nobody has actually stopped running it — that's solving a problem you don't have yet at the cost of process overhead you'll definitely pay.

Key takeaways / mental model#

A suite's value is (roughly) correctness-per-test times how-often-it-actually-runs, and speed drives the second factor much harder than most engineers intuit — a slow suite that's "still correct" can still be worthless in practice. Fix the dominant cause (unnecessary real I/O) with test doubles, and architect deliberately in speed/realism tiers rather than treating "fast" as a uniform requirement for every test.

Self-check questions#

  1. Your team's suite has grown from 4 seconds to 6 minutes over a year, one small addition at a time. Walk through how you'd diagnose which tests are the dominant contributors, and what you'd check before "fixing" each one.
  2. Explain, using the tiering argument, why it would be a mistake to try to make an end-to-end test as fast as a unit test rather than simply running it less often.
  3. A "unit" test for a pricing calculator can't run without a real database, because the calculation logic is embedded inside a repository class. What does this suggest about the production code, beyond just the test?
  4. Give a concrete example of when accepting a slower, real-I/O test is the correct choice despite the cost this lesson describes.

References#

  • xUnit Test Patterns: Refactoring Test Code (Gerard Meszaros), Chapter 18: "Test Smells: Test Performance" (Slow Tests).
  • See also: xunit-test-patterns/05 for the test-double substitution that's the primary fix, and xunit-test-patterns/04 for the Fresh-vs-Prebuilt Fixture trade-off that also affects suite speed.