Blog

Visual regression testing without CI or a subscription

By · · 6 min read

In July I pointed my own extension at the home page of divan.ru, a large Russian furniture store, saved a baseline and asked for a layout check straight away. Nothing had changed on the page (I hadn't even scrolled), and the report still listed 3,349 overlapping elements, most of them marked critical. Nearly all of those overlaps were there on purpose: carousel slides stacked on top of each other, dropdown menus waiting to open, badges sitting on product photos.

That run changed how the extension works. It now reports only what's new compared with the baseline, and the same unchanged page gives 5 findings instead of 3,349 (my own measurement; all five come from a hero carousel that rotates more slowly than the extension's detection window, more on that below). On CNN the count went from 688 to 0. I'm starting with this story because noise is the main reason people give up on visual regression testing, whatever tool they use.

Why visual testing usually ends up in CI

The standard free option for developers is screenshot assertions in Playwright, toHaveScreenshot(). It works well, but its own documentation warns that browser rendering can vary with the host OS, browser version, settings, hardware and even headless mode, and it advises generating baseline screenshots in the same environment where the tests run (Playwright docs). In a team that usually means a Docker image, a CI job, test code for every page, and someone on the team who updates snapshots whenever the design changes on purpose.

Hosted services take the environment problem off your hands and charge per snapshot instead. Chromatic's free plan includes 5,000 snapshots a month, and its Starter plan is $179 a month for 35,000 (pricing page, checked on 4 November 2026). Twenty pages at four widths is 80 snapshots a run, so three runs a working day for a month comes to roughly 5,300, which already doesn't fit into the free tier. For a team that merges pull requests all day that's money well spent. For a freelancer checking a client's site before a plugin update, or a developer who wants to know whether a CSS refactor broke anything, I think that's more setup than the question needs.

A baseline in the browser

The idea behind UI Testing Inspector is to take both screenshots on the same machine, in the same Chrome, with the same fonts and the same display scaling. Most of the rendering noise Playwright warns about comes from comparing images made in different places, so when the baseline and the check are made a few minutes apart in one browser, that whole class of false positives mostly goes away, and you don't need Docker for it.

The workflow is short. Open the page (localhost, a staging site behind a login and production all work, because the extension sees whatever your browser sees), click the toolbar icon and click Take baseline. Make your change, reload, click the icon again and click Compare with baseline. The report shows elements that moved, disappeared or changed size, new overlaps and newly clipped text, outlined and numbered on the screenshot. A layout shift that pushes a whole section down is grouped into one entry (in one live test 833 shifted elements became a single finding), so you don't scroll through hundreds of rows that all describe the same cause.

Dynamic content is handled when you take the baseline. The extension takes two snapshots of the DOM 1.5 seconds apart, and anything that changed between them (a carousel, a countdown, an ad slot, a "people are viewing this" counter) is masked in both the structural and the pixel comparison. You don't have to write ignore selectors, although it also means you won't see a real regression inside a masked area. The same masks work in mockup comparison against a Figma export.

Everything stays in the browser. Screenshots and reports live in IndexedDB, the last 10 reports are kept, and you can save the baseline to a file and send it to a colleague. The extension has no server and doesn't upload anything.

How to reduce false positives in visual regression tests

Most false positives come from four places, and I deal with them in this order: a different rendering environment, content that changes on its own, animations caught mid-frame, and one inserted block shifting everything below it. The first is the cheapest to fix. Make the baseline and the check on the same machine, in the same browser and at the same display scale, and most antialiasing noise never appears. When that isn't possible (a baseline file from a colleague, say), at least make the mismatch visible, which is why the report flags a different screen scale, window width, system or set of fonts.

Dynamic content and animations come next. I don't maintain ignore lists: the two DOM snapshots taken 1.5 seconds apart find the carousel, the timer and the ad slot, and CSS animations and videos are paused while the screenshot is taken, so a button caught halfway through a transition doesn't count as a change. The last source is the noisiest. Insert a 60 px notice bar at the top and every element below it moves, so a naive diff lists hundreds of changes; matching elements with the shift is what turned 833 moved elements into one finding in the test above. It also helps to keep a structural layer next to the pixel one. Element positions and sizes don't care about antialiasing, so a pixel difference with no structural change behind it is usually rendering noise.

If you use Playwright, the same ideas map onto its options. toHaveScreenshot() already disables CSS animations, CSS transitions and Web Animations by default through the animations option, mask covers the locators you know are dynamic, and generating baselines in the same Docker image as the CI run removes most of the environment noise (toHaveScreenshot options). What it won't do is find the dynamic parts on its own, so the mask list needs updating whenever the page gets a new widget.

Where it falls short

It checks one page at a time, when you click. There's no crawler, no schedule and no status on a pull request, and if you need to check 200 pages after every deploy, a CI tool or a hosted service is the better fit. Carousels that change slides less often than about every two seconds can escape the mask and show up as changes (that's exactly what the five findings on divan.ru are), and for now the fix is to click Retake or pause the slider before you take the baseline.

I also haven't measured how much noise a baseline file made on a Mac produces when someone compares against it on Windows. My guess is that text antialiasing will show up as small pixel differences, while the structural part of the report (what moved, what disappeared) should hold, since it compares element positions rather than pixels. The report does warn you when the two snapshots were taken on different systems, with a different screen scale or different fonts.

When each approach makes sense

If your team already runs end-to-end tests in CI, add Playwright screenshot assertions there, and if you build components in Storybook, Chromatic fits naturally. If what you want is a quick before and after on a real page, without writing tests or paying per screenshot, a browser baseline is enough for most of the bugs I see in practice: a block that slid under the header, a button that wrapped onto a second line, a section that vanished after a plugin update.

Links: Playwright visual comparisons · Chromatic pricing · UI Testing Inspector FAQ · UI Testing Inspector in the Chrome Web Store

P.S. If the extension reports noise on a page where nothing changed, send me the address through the form on the site. Most of the masking rules came from pages like that, so far mostly ones I found myself.

Ana Do builds UI Testing Inspector and writes about checking layouts and measuring how reliable AI systems are. More writing and research at anaexplores.github.io.