P0 / P1 · Chapter 05 / 15

Kafka / Async / Database

The 60-second cheat sheet#

An HTTP response can acknowledge work before an event-driven system finishes it. In the illustrative verification flow: request → producer → Kafka broker → consumer → database projection → status API → UI. A 202 response proves acceptance under the contract; a final business state and its correlated effects prove completion.

Delivery, ordering and consistency#

Kafka exactly-once features have a defined transactional boundary and do not automatically make arbitrary database/external side effects exactly once. Test idempotent handlers with the same business/event ID and verify one logical effect, including after consumer restart.

Bounded polling for completion#

// tests/api/verification.spec.ts: complete module, illustrative sandbox contract.
import { test, expect } from '@playwright/test';
test('accepted verification reaches final state', async ({ request }) => {
  // Fixture-provided unique user assumed; no production accounts.
  const id = process.env.QA_VERIFICATION_USER_ID;
  if (!id) throw new Error('Unique sandbox user required');
  const path = `/api/users/${encodeURIComponent(id)}/verify`;
  const accepted = await request.post(path);
  expect(accepted.status()).toBe(202);
  let observed = 'UNOBSERVED';
  await expect.poll(async () => {
    const response = await request.get(`/api/users/${encodeURIComponent(id)}/verification`);
    expect(response.status()).toBe(200);
    const body: unknown = await response.json();
    if (!body || typeof body !== 'object' || !('status' in body) || typeof body.status !== 'string') {
      throw new Error('Invalid status response');
    }
    observed = body.status;
    return ['VERIFIED', 'REJECTED'].includes(observed);
  }, { timeout: 20_000, intervals: [200, 500, 1_000], message: 'verification terminal state' }).toBe(true);
  expect(observed).toBe('VERIFIED'); // REJECTED is terminal but fails immediately here
});

This poll stops on either known terminal outcome, then asserts success. Exceptions inside poll may be retried; a malformed/unavailable response is reported by the final poll failure and should include diagnostics. If unknown states must abort immediately, use a dedicated bounded polling helper with explicit error classification. Set test budget above the poll plus setup/teardown. Do not repeat the initiating POST in the polling callback.

End-to-end evidence plan#

import type { APIRequestContext } from '@playwright/test';
import { performance } from 'node:perf_hooks';
export async function waitVerified(api: APIRequestContext, path: string, timeoutMs = 15_000): Promise<void> {
  const deadline = performance.now() + timeoutMs;
  while (performance.now() < deadline) {
    const remaining = Math.max(1, Math.floor(deadline - performance.now()));
    const response = await api.get(path, { timeout: remaining });
    const code = response.status();
    if (code !== 200 && code !== 503) throw new Error(`Fatal status read: HTTP ${code}`);
    if (code === 200) {
      const body: unknown = await response.json();
      if (!body || typeof body !== 'object' || !('status' in body)) throw new Error('Invalid status schema');
      if (body.status === 'VERIFIED') return;
      if (body.status !== 'PENDING') throw new Error(`Terminal/unknown state: ${String(body.status)}`);
    }
    const delay = Math.min(500, Math.max(0, deadline - performance.now()));
    await new Promise<void>(resolve => setTimeout(resolve, delay));
  }
  throw new Error('Verification completion deadline exceeded');
}

1. Provision an authorized unique sandbox user and correlation ID; stub external identity/payment side effects.
2. Submit verification and assert 202 plus accepted operation ID if documented.
3. Through public status API poll to the contract's terminal state; verify the UI reflects that same user's result.
4. With authorized hooks, inspect Kafka key/event ID/schema and consumer/projection logs; check DB row and audit record for the same correlation.
5. Test duplicate delivery, delayed consumption, rejection, consumer recovery and dead-letter handling separately.
6. Clean records or use documented TTL; retain sanitized failure evidence.

Public API completion proves externally observable behavior, not exactly which internal consumer ran. Kafka UI or DB access requires approved test harness permissions; do not pretend a browser test automatically observes internal events. A UI heading alone proves neither accepted event nor persisted result.

Reconciliation and ETL validation#

After-write reads can race with transaction commit or read replicas. Retry only where eventual consistency is expected; retrying an invariant on a strongly consistent endpoint may hide a bug. For multi-query reconciliation, choose a stable snapshot/watermark or allow a documented lag window. Align time zones and late-arriving events before classifying missing data. FINA ETL/data integrity experience provides a grounded story; add only true ownership/results.

Observability and six scenarios#

Interview response#

30 seconds: “I distinguish transport acceptance from final business state. I correlate the request, event, projection and API result, poll a meaningful terminal state with a bounded deadline, and test duplicates and ordering according to the actual Kafka contract. For reconciliation I compare keys and values as well as counts.”

Memory hook: accepted is a beginning; correlate, converge, reconcile.

Quick practice · reveal answers#

Official references and related topics#

Optional online further reading; all core study material is included above.

• Kafka design
• Polling assertions
• PostgreSQL isolation

Back to library index