P0 · Chapter 08 / 15

Debugging / Flakiness / Reliability

The 60-second cheat sheet#

A flaky test has inconsistent outcomes without a corresponding relevant code/environment change explaining the expectation. Investigate the whole system: product, test, infrastructure, data and timing. A retry that passes is evidence to classify, not a repaired test.

Reproducible diagnosis#

Capture commit, test name, seed/data identity, browser/runtime, worker/shard, timestamp and correlation ID.
Reproduce the exact CI command; compare isolated and parallel runs without destroying evidence.
Inspect error/stack, trace action/DOM snapshots, network timings, screenshots and correlated service logs.
Classify the layer and form competing hypotheses. Change one variable, such as workers=1, to test causality.
Fix the demonstrated cause, add a meaningful regression check and repeat under the previously failing conditions.
Track flake/retry trends and diagnose time; do not claim zero risk from a handful of passes.

Failure class	Evidence	Candidate fix
Product	Correct request but wrong persisted response	Product defect and regression
Automation	Listener starts after response / wrong locator	Correct synchronization/selector
Environment	Runner OOM, unavailable dependency	Capacity/health/controlled infra retry
Data	Same account across shards	Scoped provisioning/leases
Timing	Event not projected within expected contract	Poll completion; investigate latency objective

Tools and diagnostic configuration#

bash

npx playwright test --project=chromium --workers=1 --repeat-each=10
npx playwright test --debug
npx playwright show-report
npx playwright show-trace test-results/example/trace.zip

typescript

// Scoped config fragment: merge into defineConfig({...}).
const diagnostics = {
  retries: process.env.CI ? 1 : 0,
  use: { trace: 'on-first-retry' as const, screenshot: 'only-on-failure' as const,
    video: 'retain-on-failure' as const },
};

Trace Viewer answers which action was waiting, what DOM existed and which requests occurred. It does not automatically include every backend event. test.step gives business labels; stack traces locate failed expectations. First-retry tracing may miss an unreproducible first attempt, so temporarily enable tracing for every attempt during focused investigation with controlled retention.

Failure patterns and fixes#

Wrong response to failure	Evidence-based correction
Raise every timeout from 30s to 120s	Identify waiting operation and measured legitimate latency; set a scoped budget
click then listen	Register waitForResponse before click
Read innerText once for eventual state	`await expect(locator).toHaveText(...)`
Fixed 5-second sleep	Poll final business state with deadline
Single user in every shard	Unique account scope across pipelines
Cleanup only after passing assertions	Fixture teardown/finally; retain cleanup errors
Only `expect(response.ok()).toBeTruthy()`	Exact status, shape and required state/effect
Skip intermittently failing critical flow	Track quarantine risk, replacement coverage and repair owner

typescript

// Scoped test body: prevent a race and check both transport and UI.
const pending = page.waitForResponse(r =>
  new URL(r.url()).pathname === '/api/settings' && r.request().method() === 'PUT');
await page.getByRole('button', { name: 'Save', exact: true }).click();
const response = await pending;
expect(response.status()).toBe(200);
await expect(page.getByTestId('save-status')).toHaveText('Saved');

Other causes: unstable accessible names, overlays intercepting clicks, asynchronous UI hydration, stale cookies, shared DB records, high worker resource pressure, browser differences, locale/time zone, network instability and Kafka consumer lag. Replacing selectors or adding force without evidence can mask actual usability defects.

Worked RCA: 200 local passes, 30 CI timeouts#

Observation: local isolated test passed 200 times; a parallel CI batch had 30 intermittent verification timeouts. These are hypothetical diagnostic counts, not candidate career metrics. Local repetition alone did not reproduce CI concurrency.

Hypothesis	Evidence to collect	Decision
Response listener too late	Trace shows PUT response before listener wait starts	Register listener first; reproduce with fast response
Shared session invalidation	Earlier shard gets 401 immediately after later login	Unique account lease; assert auth health
Projection delay	202 then PENDING, consumer lag and later VERIFIED	Bounded poll using service objective; investigate lag separately
Runner saturation	Long actions + CPU/memory pressure, improves at fewer workers	Tune capacity and workers; retain coverage
Wrong row	Multiple matches / same display name	Scope by unique entity identity

Suppose logs show the same account in all shards and revocation at each login. The targeted fix is shard/worker-safe account leasing. Re-run the same parallel configuration, overlapping pipelines and an intentional password-change isolation case. Report observed failure rate with run count/window and remaining uncertainty; do not generalize “200 passes” to proof of universal reliability. If another cause remains after account isolation, continue classification rather than declare every timeout solved.

Metrics and interview response#

Track flaky rate with a defined denominator (tests that change outcome across attempts / executed tests), retries consumed, duration p50/p95, failures by category, false-positive rate, defect leakage, mean time to diagnose and CI trends. Pass rate alone can improve by deleting useful tests. Compare coverage and fault detection too.

30 seconds: “I preserve CI evidence, reproduce the failing execution conditions and classify product, automation, environment, data or timing causes. I inspect traces and correlation logs, test competing hypotheses and apply a targeted fix. I measure the same parallel workload afterward; increased timeouts or retries alone are not an RCA.”

90-second senior answer: handling a recurring CI flake

I start with a failure timeline rather than a guess. I capture runner/browser versions, test data, account, worker and shard, then compare the last successful action with network and service logs. If reducing workers changes behavior, that narrows the problem but doesn't prove CPU saturation: shared accounts can also be concurrency-sensitive. I test those hypotheses separately. For a late listener I fix ordering; for eventual consistency I poll final state; for revoked auth I change allocation. I retain a controlled regression that exposes the original failure mode. If quarantine is necessary, I name the missing risk coverage and arrange replacement evidence, an owner and an expiry. I report confidence based on run counts and environment, not an absolute guarantee.

Memory hook: preserve → classify → falsify hypotheses → fix → measure.