Our software factory opens a lot of pull requests. AI agents work in parallel, each on its own change, and every change goes through the same checks before it can merge. That is the rule, and it does not bend.
The trouble is what “the same checks” used to mean: everything, every time. A one-line fix to a report ran the whole test suite, the security scans, the container build and the isolation tests. On one of our products, that was about 11 minutes of build-machine time per pull request, and the same again when it merged. The cost grew with the size of the codebase, not with the size of the change.
So we taught the pipeline to ask a simple question first: what could this change break?
Step one: a plan before any test runs
Every run now starts with a small planning job. It installs nothing and finishes in seconds. It looks at the files the change touches and decides which jobs run and which tests they run. Every other job reads that plan.
The plan is a plain script with its own tests. We looked at ready-made tools first. We chose our own because we wanted the plan itself to be testable: when it makes a decision, there is a test that says why.
Step two: follow the imports
The script reads every source and test file and builds a map of who imports whom. Then it walks that map backwards from the changed files. A test is selected when it sits within two import steps of a change, or when the test itself changed.
Two steps, not one. We started with one, and a real miss taught us better: a change reached a test through a second file, the pull request passed, and the full run after the merge caught it. The fix was not a one-off exception. The depth went to two, and that exact case became a test of the selector.
Step three: the dependencies imports can’t see
Imports don’t tell the whole story. Some tests read files by path: database migrations, prompt files, seed data, recorded API responses, fixtures. No import connects them, so the map alone would miss them.
For those, a small rules file maps each path to the tests that read it. And because a list like that rots, the selector has guard tests. If a test starts reading a fixture folder that doesn’t select it, the build fails. If the test setup imports a file the selector doesn’t treat as global, the build fails. The safety net checks its own holes.
Step four: jobs that turn on only when they matter
Tests aren’t the only cost. The plan also decides the other jobs:
- Security scans run on the changed files when a change touches login, encryption, customer isolation or the web edge.
- The container build and scan run when the change touches the build itself: the Dockerfile, dependencies, compiler settings.
- Dependency audits run when the package files change.
- Lint and formatting check the changed files only.
A few checks run on every pull request no matter what: type checking, the customer isolation suite, the secrets scan and the planning job itself.
The rule that keeps it safe: unknown means everything
The most important line in the selector is the fallback. If a changed file isn’t covered by any rule, the plan runs the full suite. Global files, like the test configuration or the dependency list, also trigger a full run after the merge. So does any database migration, a force push, or any doubt about what the last good run was.
Speed is never bought with a guess. When the plan doesn’t know, it runs everything.
Behind the pull request: three more nets
Selecting tests on a pull request is only safe because more checks sit behind it.
- After a merge, the base is the last green run, not the previous commit. If an earlier run failed or was replaced by a newer one, its changes are selected again. Nothing slips through because a run got skipped.
- Every night, everything runs. The full suite, coverage floors, AI evaluations and mutation testing, which changes the code on purpose to check that a test notices. The nightly never deploys anything. It only reports.
- A red night opens one issue. It names the commit, the last good run and the failing step, with a recipe: rerun the failing test five times to tell a flaky test from a real regression, then bisect. If the selector missed a dependency, the fix goes into its rules, with a test. A flaky test gets fixed, never retried until it passes.
And nothing deploys from a cancelled or failed job. A deploy checks each job’s result explicitly.
What it changed
The honest numbers depend on the size of the change.
- Small changes win big. For pull requests that touch five files or fewer, the median run selects under 3% of the test files. One single-file change we measured took about 2 minutes instead of 11.
- Big changes still run most of the suite. A pull request that bundles a feature across 35 files reaches the shared core, and the core is imported by almost every test. About 70% of the tests run. That is correct: a change that wide deserves a wide check.
- The core decides how narrow selection can get. Splitting large files barely moved the numbers. What moves them is fewer “hub” modules that every test imports. So the selector changed how we write code: file size limits, no import cycles, and helpers that build only the part of the app a test needs.
What we tried and set aside
We also built a finer map from line-level test coverage: run every test once, record which lines it executes, select by line. It took six minutes to build and, on the same nine pull requests, selected more tests than the import rule did, 901 against 749. Shared test setup runs half the app for every database test, so every test “touched” almost everything. More precise on paper, wider in practice.
We also tried more parallel test workers on small build machines. Each worker gets its own copy of the test database, so tests never share rows. On a 4-core laptop it cut a run from about 6 minutes to 2. On a 2-core build machine it barely helped. CPU-bound tests and the database just fought over the same two cores.
Why this matters to our customers
You will never see this pipeline. What you see is the result: small fixes reach you faster, because they don’t wait behind a full test run they don’t need. And the bigger changes still get the full treatment, every night, before anyone calls the main line green.
Fast where it is safe. Thorough where it counts. That is the same rule we apply to the software we build for you.
If your team is fighting slow builds or flaky tests, we can help.