Flaky Tests: The Real Cost to Engineering Velocity (2026)
September 23, 2026
Amir Tavafi
12 min read
Flaky tests are automated tests that pass and fail on the same code with nothing changed in between, and most engineering teams treat them as background noise instead of the velocity problem they actually are. Abloomify's CI/CD pipeline health view tracks flaky-test rate next to DORA metrics and PR cycle time, in the same platform, because a test nobody trusts is a merge nobody fully trusts either.
Key Takeaways
Q: What is a flaky test?
A: A flaky test passes and fails on the same code with no changes in between, usually because of race conditions, order dependency, or an external resource like a database or API call. Abloomify's CI/CD health view tracks flaky-test rate per repo alongside DORA metrics, so a rising rate shows up before it erodes trust in the whole pipeline.
Q: What causes flaky tests?
A: Four root causes cover most cases: race conditions in async code, dependence on external resources (databases, APIs, the network), tests that aren't isolated from each other's state, and order dependency, where a test only passes if it runs after another one. Rarely is a flaky test actually random.
Q: How do flaky tests hurt engineering velocity?
A: Every flaky test adds a rerun, a pause while someone decides whether the failure is real, and eventually a quiet erosion of trust in the whole suite. Multiply that across a team merging dozens of PRs a day and the time adds up fast, without ever showing up as its own line item.
Q: How do you detect flaky tests?
A: Track pass and fail history per test across repeated runs, not just the latest result. A test that flips outcome on identical code across the last 10 to 20 runs is flaky. Abloomify surfaces this as part of CI/CD pipeline health, next to change failure rate and PR flow, instead of in a separate CI dashboard nobody checks daily.
Q: Can Abloomify detect flaky tests?
A: Yes. Abloomify's engineering intelligence layer includes CI/CD pipeline health with flaky-test detection, computed straight from GitHub alongside all four DORA metrics (deployment frequency, lead time, change failure rate, MTTR), each banded Elite to Low.
What is a flaky test?
A flaky test is an automated test that produces different results, pass one run, fail the next, against the exact same code and the exact same test environment, with nothing in the codebase changed in between. Unlike a genuinely broken test, which fails consistently until someone fixes the underlying bug, a flaky test's failure carries no reliable signal: rerun it and it might pass, changing nothing except the clock. That unpredictability is what makes flaky tests expensive in a way a normal bug isn't. A team can triage a consistent failure in minutes because the failure means something. A flaky failure means someone has to stop, guess whether it's real, usually rerun the pipeline, and move on having learned nothing about the actual state of the code. Multiply that across a CI pipeline running dozens of times a day and flaky tests turn into a tax on every merge, whether or not that test ever catches a real bug.
Engineering teams usually notice flaky tests long before they name them. It's the test everyone reruns without reading the output, the one with an old
// TODO: fix flakiness comment nobody's touched in months, the CI badge that's red so often nobody trusts red anymore. That last part is the real damage: once a team stops trusting CI failures, people start merging past them, which is exactly when a real regression slips through disguised as "probably just that flaky test again."
Why flaky tests happen
Flaky tests almost never happen for genuinely random reasons, even though they feel random from the outside. In practice, the overwhelming majority trace back to four root causes: race conditions in asynchronous code, where timing rather than logic determines the outcome; dependence on an external resource like a live database, API, or network call that isn't fully controlled in the test environment; tests that aren't properly isolated from shared state, a database row, a global variable, a file on disk, left behind by a previous test; and order dependency, where a test only passes when it happens to run after another specific test that silently primes some state it relies on. Knowing which of the four is responsible changes the fix completely, which is why "just rerun it" as a permanent strategy guarantees the flakiness never actually goes away.
- Race conditions: async code that doesn't wait for a resource, just hopes it's ready in time.
- External dependencies: a live API, a shared staging database, a clock, anything the test doesn't fully control.
- Poor isolation: two tests reading or writing the same database row, cache key, or global state.
- Order dependency: a test that silently needs a different test to run before it.
The hidden cost of flaky tests on engineering velocity
Flaky tests cost engineering teams time in a way that rarely shows up as a single line item, which is exactly why they survive quarter after quarter on backlogs that never get prioritized. Every flaky failure adds a rerun burning CI minutes, a pause while an engineer decides whether the failure is real, and, if it happens often enough, a quiet erosion of trust in the whole test suite that eventually shows up as engineers merging past red builds because "it's probably just that test again." None of that appears on a sprint burndown. It shows up instead as PR cycle time creeping up, change failure rate drifting worse because a real regression got waved through as noise, and a CI pipeline that technically runs green but that nobody actually believes.
How to detect flaky tests
Detecting a flaky test means tracking its pass and fail outcome across repeated runs on identical code, not judging any single run in isolation, since the entire definition of flaky is inconsistency across runs that should be identical. The simplest detection method is a rerun-on-failure policy: if a test fails once and then passes on an immediate rerun with no code change, flag it as a flaky candidate rather than treating the first failure as gospel or the second pass as proof of health. A more reliable method tracks pass rate per test over a rolling window, the last 20 or 50 runs, and flags anything below roughly 98 to 99 percent consistency, since a genuinely stable test should almost never flip outcomes on unchanged code. CI providers increasingly bake this in, and engineering intelligence platforms that already read CI history can surface the same signal without a separate login.
Where detection breaks down is scale. A five-person team can informally track which tests are flaky in Slack threads and tribal memory. A fifty-engineer org across a dozen repos can't. The flaky test one team fixed quietly reappears in another repo nobody's watching, and the signal that actually matters, whether flaky-test rate is trending up or down this month, gets lost because nobody has one place to look.
How to fix and eliminate flaky tests
Fixing a flaky test starts with quarantine, not root-cause analysis: pull the test out of the blocking suite the moment it's confirmed flaky, tag it or move it to a non-blocking job, so it stops corrupting the signal for every other PR while someone investigates on a normal timeline instead of under merge pressure. From quarantine, root-cause each flaky test against the four categories above. Race conditions get fixed with explicit waits or synchronization instead of arbitrary sleeps. External-resource flakiness gets fixed with mocks, stubs, or a properly isolated test environment instead of a shared staging database three teams write to at once. Isolation problems get fixed by resetting state between tests instead of assuming a clean slate. Order dependency gets fixed by making every test runnable alone, which most test runners can verify directly by randomizing run order in CI.
- Quarantine on confirmation, not on suspicion. Move it out of the blocking path immediately.
- Assign an owner and a deadline. A quarantined test with no owner becomes permanent.
- Fix the root cause, not the symptom. A longer timeout hides a race condition; it doesn't fix one.
- Track flaky-test rate as a metric, not a one-time cleanup. New flaky tests appear as fast as old ones get fixed if nobody's watching the trend.
- Budget for it. Teams that treat flaky-test cleanup as "when we have time" never have time.
For a deeper technical treatment of the underlying causes, Martin Fowler's writeup on eradicating non-determinism in tests is still one of the clearest references in the industry.
How Abloomify puts flaky-test rate next to the metrics that depend on it
Abloomify's engineering intelligence layer reads CI/CD pipeline health, including flaky-test detection, straight from GitHub alongside all four DORA metrics (deployment frequency, lead time, change failure rate, MTTR), each banded Elite through Low, so a rising flaky-test rate shows up next to the change failure rate and PR cycle time it's actually driving instead of living in a separate CI dashboard or a standalone point tool most engineers never open. A dedicated flaky-test tool can tell you a test is flaky. It can't tell you that the same repo's change failure rate started drifting the same week, or that reviewers stopped trusting green checkmarks and PR cycle time crept up as a result. That's the same argument Abloomify makes about security posture and EPSS scores: a signal read in isolation tells you less than the same signal read next to the delivery data it's actually affecting.
Big companies bolt on a separate tool for every symptom: one for flaky tests, one for DORA, one for security posture, one for PR flow. We built one platform that reads all of it off the same GitHub connection, PII-free, so a VP of Engineering isn't stitching four dashboards together to answer why cycle time crept up this month.
Choosing a platform that treats flaky tests as a metric, not a mystery
Evaluating how to handle flaky tests starts with a simple question: is flaky-test rate something your team can see as a trend, or is it something you only notice when someone complains in standup? A dedicated test-execution tool can quarantine flaky tests inside your test runner, which matters, but it can't tell you whether the fix mattered at the level a VP of Engineering actually reports: did change failure rate improve, did PR cycle time come down, is the DORA band still Elite. Abloomify connects to GitHub, GitLab, and Bitbucket alongside Jira and Linear, PII-free and SOC 2 Type II certified, and reads CI/CD pipeline health, flaky-test rate included, into the same Engineering Velocity Score as DORA metrics, PR flow, security posture, and AI tool ROI, so a rising flaky-test rate is a line on a dashboard someone already checks instead of a fact someone has to go looking for.
A test nobody trusts might as well not run. Fix the signal, and the pipeline starts telling the truth again.
FAQ
What is a flaky test?
A flaky test passes and fails on the same code with nothing changed in between, usually because of a race condition, an external dependency, poor test isolation, or order dependency. Abloomify tracks flaky-test rate per repo as part of CI/CD pipeline health, next to DORA metrics, so a rising rate shows up before it erodes trust in the whole pipeline.
What causes flaky tests?
Four root causes cover most cases: race conditions in asynchronous code, dependence on an external resource like a database or API the test doesn't fully control, tests that aren't isolated from shared state left behind by another test, and order dependency, where a test only passes after a specific test runs before it. Flaky tests are rarely actually random.
How much do flaky tests actually cost a team?
There's no single industry number worth quoting, because it scales with flaky-test rate and team size. What's consistent is that the cost never shows up as its own line item. It shows up as PR cycle time creeping up, change failure rate drifting worse, and engineers who stop reading CI output because half of it is noise.
Should I quarantine a flaky test or fix it immediately?
Quarantine first, always. Pull the confirmed flaky test out of the blocking suite the moment it's flagged so it stops corrupting the signal for every other PR, then root-cause it against the four common causes on a normal timeline instead of under merge pressure. A quarantined test still needs an owner and a deadline or it becomes permanent.
Does Abloomify replace dedicated flaky-test tools?
No, and it isn't trying to. Abloomify reads flaky-test rate from your GitHub CI history as part of CI/CD pipeline health and puts it next to DORA metrics, PR flow, and security posture in one Engineering Velocity Score. It's the visibility layer that tells you the rate is rising and in which repo, not a test-runner plugin that quarantines tests for you.
Amir Tavafi
Co-Founder & CEO
Product leader and innovator with over 15 years of experience in the tech sector, grounded in AI and robotics. Previously led product development in fraud detection and AI solutions at Nasdaq Verafin.