Manual UAT Doesn't Scale: Building an Automated Test Suite That Actually Replaces It

In the last post I walked through rolling out a strict Content-Security-Policy on a legacy WebForms app. What I didn't cover is how the rollout was actually verified before each stage — because "click through the app and see if anything looks broken" stops being a viable strategy the moment you have more than a handful of pages, more than one browser to care about, and more than one release to get through. This post is about the other half of that project: building an automated test suite that did the job manual UAT was never going to do reliably.

Why manual UAT breaks down

Manual testing isn't wrong, exactly — it's just a strategy that scales linearly with everything you don't want it to scale with. Every additional page is more clicking. Every additional browser is another full pass. Every regression cycle after a follow-up fix is the whole thing again, from the top, because you can't be sure which parts of the app the fix might have touched. And the failure mode isn't dramatic — nobody misses the login page. What gets missed is the report nobody opens except on the first of the month, the workflow only one user in the org actually uses, the console warning that doesn't stop the page from rendering so nobody notices it.

For something like a CSP rollout specifically, this problem is worse than usual, because the thing you're testing for is largely invisible. A blocked inline script often fails silently — the button just doesn't do anything, or a dialog doesn't appear — and a tester clicking through the happy path has no reason to suspect the cause is a security header rather than a bug. You need something that's actually looking at the browser console on every single page, every single time, which is not a realistic ask of a human being doing it by hand.

Design principle: tests map to acceptance criteria, not to "test everything"

The temptation with test automation is to try to cover the whole app exhaustively, which just recreates the manual-testing scaling problem in code instead of in human hours. The better approach is to go back to whatever acceptance criteria the change was actually signed off against, and write one test file per criterion, numbered to match:

tests/
  ac1-headers.spec.js       // AC1: CSP header present, correctly formed, correct mode
  ac2-console.spec.js       // AC2: zero console violations across every page
  ac3-auth-flow.spec.js     // AC3: login / OTP / session timeout still work
  ac4-dialogs.spec.js       // AC4: confirm dialogs and modals still fire
  ac5-exports.spec.js       // AC5: Excel/PDF export still produces valid output
  ...

This does two things at once. First, it keeps the suite scoped to what was actually promised, rather than open-ended "test coverage" that never feels finished. Second — and this matters more than it sounds like it should — it makes the suite legible to people who didn't write it. When a QA lead or a product owner asks "did we verify X," you can point at ac4-dialogs.spec.js directly instead of explaining which of forty test cases happens to cover that behaviour.

Three categories of check, and why you need all three

Structural checks. Does the response actually carry the header you think it does, with the directives you expect, in the right mode? This is the cheapest kind of test to write and the first thing that should exist — a single HTTP request and an assertion on the response headers, no browser needed.

const response = await page.goto(url);
const csp = response.headers()['content-security-policy'];
expect(csp).toContain("script-src 'self'");
expect(csp).not.toContain("'unsafe-inline'");

Passive monitoring — the console crawler. This is the pattern that does most of the actual work. Attach a listener to the browser's console and page-error events, then navigate to every page in the app in turn, and assert that nothing violation-shaped showed up.

const violations = [];
page.on('console', msg => {
  if (msg.text().includes('Content Security Policy')) {
    violations.push({ url: page.url(), text: msg.text() });
  }
});

for (const url of allAppPages) {
  await page.goto(url);
  await page.waitForLoadState('networkidle');
}

expect(violations).toEqual([]);

This single pattern generalizes well beyond CSP — it's the right shape for any "make sure we didn't break anything else" check, because it doesn't need to know in advance what could go wrong on any given page. It just watches, on every page, all the time, which is exactly the thing a human tester is worst at doing consistently.

Behavioural checks. The console crawler tells you nothing broke passively; it won't tell you whether the confirm dialog on the delete button still actually confirms, or whether the OTP flow still completes. For that you need real interaction tests — click the button, assert the dialog appeared, fill the form, assert the redirect happened. These are slower to write and slower to run, so keep them scoped to the flows that genuinely matter (auth, anything destructive, anything revenue-adjacent) rather than trying to interaction-test the whole app.

A headless browser, not just an HTTP client

It's worth being explicit about why a tool like Playwright (or Puppeteer) is the right choice here over a plain HTTP-assertion library like a request-based test client. The structural checks don't need a browser — but the console crawler does, because CSP violations (and most other JS-level runtime problems) only surface when the page actually executes in something that behaves like a real browser, parses the DOM, and runs the scripts. An HTTP client that just fetches and inspects markup will never see them. If your rollout involves anything that only breaks at runtime — and most interesting regressions do — you need a tool that runs a real rendering engine, not one that treats the response as a text blob.

Automated tests don't replace sign-off — they change what sign-off looks like

One thing worth building alongside the test suite itself: a lightweight, human-facing status view that doesn't require anyone to read code or run a test command to understand where things stand. In this project that took the form of an admin-gated in-app page that ran a handful of live checks against the current page context and rendered a plain pass/fail summary. The point isn't redundancy with the Playwright suite — it's that a formal UAT sign-off usually needs a non-engineer to look at something and say "yes, this is acceptable," and handing that person a terminal output isn't a reasonable ask.

The split that works well: the automated suite runs on every change and catches regressions continuously, cheaply, and without waiting on anyone's calendar. The human-facing view exists for the moment someone actually needs to sign a form saying they checked it. Neither one replaces the other — the mistake is thinking automation means you no longer need a human in the loop; it just means the human's time gets spent on the judgment call ("is this acceptable to ship") instead of the mechanical part ("did anything break").

Practical notes from actually running this

• Run structural and console checks against every rollout stage separately. If you're doing a staged rollout (report-only, then enforced, or any equivalent staged config), run the full suite against each stage as its own pass. A policy that's clean in report-only mode can still surprise you once it's actually enforcing, because report-only never blocks anything — it only tells you what would have been blocked, and passive reporting has its own blind spots.
• Expect flakiness from network-idle waits, not from your assertions. Most of the maintenance burden in a suite like this isn't the checks themselves — it's timing. Pages with polling widgets or long-lived connections never truly go network-idle, so lean on explicit "wait for this specific element" conditions over blanket idle waits where you can.
• Keep the page list in one place, not scattered across spec files. A shared config listing every route the crawler needs to hit means adding a new page to the app is a one-line addition to the test suite, not a "did anyone remember to add coverage" question.
• Treat an empty violations array with suspicion the first time you see it. If your console crawler comes back completely clean on the very first run, check that the listener is actually wired up before you celebrate — a silently-broken listener and a genuinely clean app produce the same output.

The takeaway

Manual UAT isn't obsolete, but it should be reserved for the parts of verification that actually require human judgment — does this feel right, is this an acceptable tradeoff, would a real user be confused here. Everything mechanically repeatable — did the header show up, did the console stay quiet, did the known flows still complete — belongs in a suite that runs the same way every time, doesn't get tired on page forty, and doesn't need to be re-run by hand every time someone touches an unrelated file. Build the automated layer first, scoped tightly to what you actually promised to verify, and manual testing stops being the bottleneck it currently is.