A flaky build passes and fails on the same code. It is one of the most corrosive problems in engineering, because it destroys trust in the signal the pipeline exists to provide. Once people reflexively click “re-run”, real failures hide among the false ones. This guide gives you a method for diagnosing flakiness rather than just retrying around it.

Why Flakiness Is Worse Than It Looks

Each flaky failure costs minutes of waiting and attention, but the larger cost is behavioral. Developers learn that red does not mean broken, merges slow down, and genuine regressions slip through because “it’s probably just that test again.” Treat flakiness as a defect with an owner, not as weather.

Step One: Confirm It Is Flaky, Not Broken

First rule out a consistent failure that only appears under certain conditions. Re-run the same commit several times and note the pattern. Does it fail on every run on one runner type but never on another? Only on the first run after a cache clear? Only at certain times of day? Patterns turn a vague complaint into a hypothesis.

Record each failure with the commit, the job, the runner, the duration and the error message. A simple shared log or spreadsheet beats memory, and it lets you rank which tests or jobs cause the most pain.

Step Two: Classify the Failure

Most flakiness falls into a handful of families. Naming the family narrows the search quickly.

Timing and Concurrency

Tests that rely on sleeps, fixed timeouts or an assumed order of asynchronous events pass when the machine is fast and fail when it is loaded. Race conditions in the code under test show up the same way. Replace fixed waits with waiting for an explicit condition, and fix real races in the application rather than the test.

Shared or Leaked State

Tests that depend on leftover data, global variables, files or ports from earlier tests may pass in one order and fail in another. Run the suite in random order and in isolation to expose this. Each test should create and clean up what it needs.

Environment and Infrastructure

Runners can differ in CPU, memory, disk space, installed tool versions and network conditions. Resource exhaustion, out-of-memory kills and exhausted disk are classic causes of unexplained mid-job failures. Pin tool versions and check runner resource metrics.

External Dependencies

Tests that call real third-party services, download packages at build time or rely on public registries inherit those systems’ reliability. Use mocks or local fakes for unit tests, and mirror or cache dependencies for builds.

Nondeterministic Inputs

Randomness, current time, time zones, locale and unordered collections can all produce occasional failures. Seed random generators, inject clocks and sort results before asserting.

Diagnosis Cheat Sheet

Symptom	Likely Family	First Thing to Try
Fails only under load or on slow runners	Timing	Replace sleeps with condition waits
Passes alone, fails in the full suite	Shared state	Run in random order, isolate the test
Fails on one runner type only	Environment	Compare resources and tool versions
Error mentions network or timeouts	External dependency	Mock the call or cache the dependency
Fails near midnight or on certain dates	Nondeterministic input	Inject a fixed clock
Process killed without a stack trace	Resource exhaustion	Check memory and disk limits

Step Three: Reproduce It Locally or in Isolation

A failure you can reproduce is half fixed. Try running the single test in a loop many times, with parallelism enabled, under artificial CPU load, and in a container that matches the runner image. If it never fails in isolation but fails in the full suite, suspect shared state or resource pressure.

Step Four: Quarantine, Then Fix

You cannot always fix a flaky test immediately, but you should not leave it blocking everyone. Move it to a quarantine set that still runs and reports but does not gate merges, and create a ticket with an owner and a deadline. The deadline matters: quarantine without follow-up is just deletion with extra steps.

Automatic retries can be a useful temporary measure, particularly for known infrastructure hiccups, but they hide the problem. If you use them, track the retry rate so flakiness stays visible.

Step Five: Prevent Recurrence

• Run new tests multiple times before merging them.
• Run the suite in random order regularly.
• Keep tests small, independent and fast.
• Capture logs, screenshots or artifacts on failure so the first occurrence is diagnosable.
• Review flaky-test metrics weekly and treat the worst offenders as priorities.

A Realistic Example

Suppose a team’s integration job fails roughly one run in several, always in a test that checks a record was created. Logging reveals the test reads from a database replica immediately after writing to the primary, and replication occasionally lags behind the read. The failure is a genuine race in how the test (and arguably the application) handles read-after-write consistency, not a random glitch. The fix is to read from the primary within that flow, and to make the test wait on an explicit condition. A retry would have hidden a real design assumption.

Frequently Asked Questions

Should we just add automatic retries to every job? Retries are a pressure valve, not a cure. Use them narrowly, track how often they fire and keep working on root causes.

How do we decide which flaky test to fix first? Rank by how often each fails and how many people it blocks. A moderately flaky test in a critical path usually beats a very flaky test nobody depends on.

Is deleting a flaky test acceptable? Sometimes, if it no longer protects meaningful behavior. Otherwise, rewrite it. Deleting coverage silently is the worst option.

Can slow runners cause flakiness? Yes. Timing-sensitive tests are often the first to reveal underpowered or overloaded runners.

Who owns flaky tests? The team that owns the code under test, with the platform team owning shared infrastructure causes.

Next Step

Start a simple flaky-failure log this week. After two weeks, sort it by frequency, pick the top offender and walk it through the classification above. One fixed test teaches the team more than a dozen discussions.