Flaky Test Triage
A testing loop for failures that do not reproduce consistently.
Use loop copies the kickoff prompt. Copy Markdown and Download loop include the complete Markdown page with context, source attribution, anchors, and guardrails. Cursor / Claude Code buttons prepare the shorter kickoff prompt only.
Supported agents
Goal
Diagnose flaky behavior without hiding it.
Feedback gate
pnpm test -- --runInBandStop condition
The flake has a likely cause, a verified fix, or a documented quarantine recommendation.
Give the agent these inputs before it starts the loop. This keeps discovery bounded and prevents vague retries.
The loop assumes these commands or integrations are available. Missing tools should be reported as blockers, not ignored.
Two separate pieces: the kickoff prompt starts the loop, while the downloaded Markdown carries the complete reference page.
1. Copy or download
Use the kickoff for a fast agent run. Download the full Markdown when you need source, context, and attribution in one file.
2. Paste into the agent
Start a fresh agent session in the target repo and provide the requested project context if the loop asks for it.
3. Let it self-pace
The agent should act, check evidence, retry only when the gate fails, and stop at the stated exit condition.
The diagram shows the order. This checklist keeps only the action, command, and failure handling needed during a real pass.
1. Re-run narrowly
Run the failing test multiple times under stable conditions.
2. Check nondeterminism sources
Look for time, random, network, shared state, and order dependencies.
3. Patch or isolate
Fix synchronization/state issues or propose quarantine with evidence.
4. Verify repeatedly
Run the test enough times to support confidence.
This is the text copied by Use loop. It is intentionally shorter than the Markdown export.
Triage the flaky test: re-run narrowly, estimate reproduction, inspect nondeterminism sources, patch or document quarantine, and verify repeatedly.
Goal: Diagnose flaky behavior without hiding it.
Check command: pnpm test -- --runInBand
Exit condition: The flake has a likely cause, a verified fix, or a documented quarantine recommendation.
Max iterations: 6
Guardrails:
- Do not weaken, skip, delete, or rewrite the validation command to force success.
- Do not claim completion until the stated exit condition is actually satisfied.
- If blocked, report the blocker, evidence, and next safest action instead of gaming the metric.
- Do not delete or skip flaky tests just to make CI pass without documenting risk.Quality
85/100
Safety
90/100
Expected output
Flake classification, reproduction evidence, and fix or quarantine recommendation.
Related loops
Browse allTest Until Green
Run the project test suite, fix root causes, and keep iterating until the selected test command passes.
Kickoff preview
Run the project tests, inspect failures, fix the smallest root cause, and repeat until the test command passes. Do not weaken the tests or change the check command. Goal: Make the selected test command pass without weakening coverage. Check command: pnpm test Exit condition: The test command exits successfully with no failing tests. Max iterations: 8 Guardrails: - Do not weaken, skip, delete, or rewrite the validation command to force success. - Do not claim completion until the stated exit condition is actually satisfied. - If blocked, report the blocker, evidence, and next safest action instead of gaming the metric. - Do not delete, skip, or mark tests as todo unless explicitly requested.
CI Failure Watcher
Poll CI status, read failing job logs, and produce the smallest actionable fix plan or patch.
Kickoff preview
Monitor the PR checks, inspect failing logs, fix the smallest root cause, push, and continue until CI passes or a real blocker is reported. Goal: Turn failing CI logs into a verified fix or clear blocker report. Check command: gh pr checks --watch Exit condition: The failing CI check is fixed, or a blocker is documented with log evidence. Max iterations: 6 Guardrails: - Do not weaken, skip, delete, or rewrite the validation command to force success. - Do not claim completion until the stated exit condition is actually satisfied. - If blocked, report the blocker, evidence, and next safest action instead of gaming the metric. - Do not paste secret-bearing logs into public output. Redact tokens and credentials.
Regression Risk Audit
Map a change to likely regressions, then run or propose focused checks for those risks.
Kickoff preview
Map the changed files to affected call chains, list likely regressions, verify the top risks, and report remaining uncertainty. Goal: Identify likely regressions and verify the highest-risk paths. Check command: git diff --name-only main...HEAD Exit condition: Top regression risks are either tested, ruled out, or documented as unresolved. Max iterations: 4 Guardrails: - Do not weaken, skip, delete, or rewrite the validation command to force success. - Do not claim completion until the stated exit condition is actually satisfied. - If blocked, report the blocker, evidence, and next safest action instead of gaming the metric. - Do not overstate confidence when a risk was not actually verified.
Ship PR Until Green
Implement a scoped change, open or update a pull request, inspect CI, and continue until all required PR checks pass.
Kickoff preview
Take this branch to a green pull request. Implement the requested change, run local verification, open or update the PR, run `gh pr checks`, inspect failures, fix root causes, and repeat until every required check passes or you hit the iteration cap. Goal: Open or update a pull request and stop only when all required PR checks are green. Check command: gh pr checks Exit condition: All required pull request checks are successful and the PR is ready for review or merge. Max iterations: 10 Guardrails: - Do not weaken, skip, delete, or rewrite the validation command to force success. - Do not claim completion until the stated exit condition is actually satisfied. - If blocked, report the blocker, evidence, and next safest action instead of gaming the metric. - Do not disable required checks, edit loops to skip jobs, or remove tests to make CI green. - Do not merge the PR unless the user explicitly asked for merge authority.