P1 · Chapter 11 / 15

LLM evals / AI test quality

The 60-second cheat sheet#

An eval is a repeatable measurement of AI behavior against explicit criteria. For generated tests, “compiles,” “passes,” “covers requirements” and “detects defects” are separate outcomes. Build an approved dataset and compare versioned agent configurations under controlled conditions.

Gold dataset and evaluation harness#

Each case contains Jira acceptance criteria, approved reference behavior, edge cases, expected scenarios, controlled application fixtures and a human-reviewed baseline. Record repository/app version, dataset version, prompt/rules, model identifier, Skills/Powers versions, tool permissions, settings, seed where supported, run time/cost and generated artifact hash. Keep failures and reviewer edits, not only successful samples.

Split development examples from held-out requirements. Include unseen negative cases, multiple domains and adversarial ambiguity. Repeating exact instruction examples as the entire evaluation rewards memorization. Repeat samples when outputs are nondeterministic; compare distributions and confidence, not one lucky run.

// Standalone TypeScript: deterministic metric calculation for reviewed scenarios.
type Counts = { correct: number; generated: number; expected: number;
  covered: number; criteria: number; killed: number; validMutants: number };
export function metrics(c: Counts) {
  const ratio = (a: number, b: number): number | null => b ? a / b : null;
  return {
    precision: ratio(c.correct, c.generated), recall: ratio(c.correct, c.expected),
    criterionCoverage: ratio(c.covered, c.criteria),
    mutationScore: ratio(c.killed, c.validMutants),
  };
}
// Counts require an agreed matching rubric; null means undefined, not 100%.

Metrics and interpretation#

Controlled defect detection#

Correct behavior displays VERIFIED after successful verification. In an isolated test environment, inject a controlled response/logic mutation returning PENDING instead. A meaningful status assertion must fail; a test that checks only the “Verification” heading may still pass. That test has execution success but weak detection ability. Verify each mutant is exercised; a startup crash is not necessarily the intended defect detection.

// Complete UI-only assertion-sensitivity test; illustrative app route.
import { test, expect } from '@playwright/test';
test('controlled wrong state is visible to the assertion', async ({ page }) => {
  await page.route('**/api/verification/eval-user', route => route.fulfill({
    status: 200, contentType: 'application/json',
    body: JSON.stringify({ id: 'eval-user', status: 'PENDING' }),
  }));
  await page.goto('/verification/eval-user');
  // In a mutation campaign, the generated success test must fail on this fixture.
  await expect(page.getByTestId('verification-status')).toHaveText('PENDING');
});

The code above verifies the mutation fixture wiring and passes for PENDING; the generated success test should instead assert VERIFIED and fail against it. Do not misreport this wiring test as a mutation kill. A UI mock is assertion-sensitivity evidence; true backend mutation tests require the controlled application mutation to be deployed in the test environment. Never mutate production.

A/A, A/B and example results#

Hypothetical example, not candidate achievements:

Neither automatically wins. Classify 15 failures, severity-weight missing criteria, review faulty assumptions and compare defect detection, cost and edit effort. CI failure caused by a real application bug can be useful evidence of a stronger test.

Judges, monitoring and interview response#

Use deterministic checks for syntax, lint, schema, real execution and policy controls. Human domain experts judge business coverage and ambiguous requirements. An LLM judge can triage using a rubric but needs calibration, disagreement review and sampled audits; don't make it sole authority. A concise dashboard should show dataset/config version, denominators, coverage, fault kills, failure classes, flake trend, review effort, cost and policy violations, with links to evidence. No colorful metric display can replace that evidence.

30 seconds: “I evaluate generated tests against approved unseen requirements on a controlled app. I measure compile/execution success separately from criterion coverage, assertion quality and seeded-fault detection. I version the model, tools and instructions, compare repeated paired runs and classify failures before choosing a winner.”

Memory hook: runs ≠ covers ≠ catches defects.

Quick practice · reveal answers#

Official references and related topics#

Optional online further reading; all core study material is included above.

• Evaluation design

Back to library index