P1 · Chapter 09 / 15

Test strategy / Architecture

The 60-second cheat sheet#

A test strategy explains which risks to cover, at which layers, with what evidence and maintenance cost. Pyramid/trophy models are heuristics, not percentage quotas. Choose the lowest useful layer that proves the risk; keep real integration/E2E for failures that isolated units or mocks cannot reveal.

Layer and technique decisions#

Layer	Best evidence	Blind spot
Unit	Fast domain branches and edge cases	Deployed wiring, real dependencies
API/service	Business rules, errors, permissions	Browser rendering and full downstream path
Contract	Interface compatibility	Entire deployed workflow or semantics
Integration	Real broker/DB/service interaction	Full user experience if UI absent
Component / mocked UI	Rendering and UI states cheaply	Real backend contract and deployment
UI/E2E	Critical user journey across boundaries	Expensive, harder diagnosis, broad data needs
Accessibility/visual/performance smoke	Keyboard semantics, unintended appearance change, latency regression	Deep specialist audits/load models need additional work

Technique	Example in verification domain
Equivalence partitions	Valid/expired/malformed identity documents
Boundary values	Age threshold, amount limits, exact expiry timestamp
Decision table	Role × document validity × country policy
State transitions	PENDING → VERIFIED/REJECTED; illegal reverse transition
Pairwise	Browser/device/locale combinations; not all higher-order risks
Exploratory	Unknown behavior around retries, navigation and partial failure
Negative	Unauthorized cross-user access, invalid IDs, dependency outage
Smoke / sanity / regression	Critical readiness / focused change check / broader unchanged behavior; agree team terminology

Repairing 200 slow UI tests#

Hypothetical problem: 200 UI tests take 60 minutes, 30 are flaky. First map each test to requirements and failure history. Delete only genuine redundancy with preserved coverage. Move repeated business combinations to API/unit tests, retain critical E2E, and use mocked UI for frontend edge states. Replace UI setup with authorized factories. Fix shared accounts and synchronization before multiplying workers. Then shard balanced independent tests, tune resources and track p95 plus coverage/fault detection. Parallelizing flawed tests can make the failure rate worse.

Risk, ROI and traceability#

Prioritize probability × business impact, informed by change history, usage frequency, financial/regulatory exposure and data sensitivity. Balance confidence against runtime, environment/setup cost, maintenance and diagnostic effort. A stable high-impact flow is a strong automation candidate; a rapidly changing exploratory question may initially benefit from human investigation.

Evidence chain	Purpose
Jira acceptance criterion	Agreed expected business behavior
TestRail case/suite	Scenario and priority; suite organizes cases
Automated test ID	Executable check and layer
Run/plan and CI result	Configuration/build/time-specific execution; plans organize runs
Defect and evidence	Actual vs expected, correlation, severity, reproducibility
Coverage matrix	What is covered, how strongly, what remains manual/unverified

CI pass rate measures outcomes of selected checks, not full requirements coverage. A TestRail run can show all cases passed while an unmodeled acceptance condition is absent. Define denominators and audit assertion quality. Vendor workflows/fields vary.

Distributed architecture and nonfunctional scope#

Microservices add network faults, schema versions, eventual consistency, ownership and dependency deployment. Test backward compatibility, retry/deduplication and observability at boundaries; don't rely exclusively on fragile full-stack tests. A monolith may simplify deployment boundaries but still has DB and external-service risks. Performance smoke needs an explicit load/latency hypothesis and controlled baseline; browser duration alone is not service capacity. Accessibility includes keyboard navigation, labels, focus and error announcements. Visual baselines need stable fonts/viewport/data and human review of intentional changes.

Selenium offers broad WebDriver ecosystem/language support; Playwright offers modern browser tooling and integrated runner facilities. Cypress has its own command queue/retry semantics rather than ordinary awaited Playwright calls. WebdriverIO builds on WebDriver/BiDi ecosystem and runner integrations. Choose using platform/language/browser/mobile needs and team maintenance capability, not a universal winner. BDD/Gherkin helps when product and engineering actively collaborate on executable examples; otherwise it can become translation overhead.

Docker standardizes userspace dependencies, not every kernel/network/resource detail. Cloud staging configuration affects identity, network routes, datasets and permissions. Distinguish a controlled environment failure from a code regression.

Senior, Lead and team decisions#

Senior engineers own reliable implementation, diagnosis, reviews and risk communication. Leads additionally coordinate priorities, standards, mentoring and stakeholder alignment; exact titles vary. Frame.so QA approach/coordination is grounded experience, but do not imply unverified architect ownership.

Review tests for requirement relevance, precise assertions, data isolation, cleanup, locator contracts, secure evidence and readable failure messages. Definition of done should include agreed acceptance checks, required gates, code review, documented residual risks and ownership. Teach a teammate by pairing on one failure, explaining the causal model, then letting them implement and explain the next fix.

If asked to skip a critical CI test, explain which risk disappears, present a targeted repair or replacement/manual evidence, and seek explicit release-risk ownership through the team's process. Respect delivery constraints without silently changing the meaning of green.

Interview response#

30 seconds: “I start with business risk and map requirements to the cheapest layer that provides useful confidence. I retain critical E2E, move repetitive business validation lower, and measure coverage, fault detection and maintenance as well as speed. I make residual risk visible and mentor the team toward independently maintainable tests.”

90-second senior answer: strategy whiteboard

For a verification application I identify permission, identity outcome, event delivery, data integrity and UI usability risks. Units cover domain decisions, APIs cover rules and authorization, contracts cover interface compatibility, integration covers Kafka/DB boundaries and a few E2Es prove the deployed user journey. I map these checks to Jira/TestRail and CI evidence, with explicit unverified cases. For a slow suite I audit duplication, flaky causes and test-data setup before adding shards. I choose blocking MR gates based on critical impact, run broader matrices later, and protect secrets in reports. Review and mentoring make the approach sustainable. When delivery pressure conflicts with coverage, I present the missing risk and concrete alternatives so the accountable owner can decide transparently.

Memory hook: risk → layer → evidence → cost → owner.