Testing in a Headless Browser

For over 12 years, I’ve been developing web-based data visualizations, and automating tests in a browser environment has always been a challenge. It’s possible, but often painful and hard to maintain. Moreover, automating benchmarks in a browser environment has been even more elusive.

In this blog post, I’ll demonstrate how easy it is to set up Vite, Vitest, and Playwright to test and benchmark TypeScript code in a headless browser. This setup allows tests and benchmarks to be run from the command line and in CI, simplifying a historically complex configuration.

My Journey to Vitest

While recently working on sigma.js v3, I started to modernize all the tooling. The existing setup consisted of outdated Webpack configurations, ad-hoc scripts, and extensive boilerplate. It was challenging to maintain and evolve, with several unresolved issues, including running unit tests in a browser environment.

I spent hours searching for a modern, easy-to-set-up testing solution, to no avail. Various blog posts suggested solutions based on Puppeteer, PhantomJS, or even CasperJS, but they all seemed outdated and/or difficult to set up (and likely maintain).

Vitest for Unit Testing

Eventually, I stumbled upon Vitest Browser Mode based on Playwright, which perfectly fit my requirements. At that point, sigma.js was already depending on Playwright for end-to-end testing. I wish this solution had been easier to find, and I hope this post accelerates its discovery for others.

Vitest for Benchmarking

During my research, I also discovered Vitest’s Benchmarking feature, powered by TinyBench. It enables Vitest to run TinyBench for executing benchmarks. It looked like “just a shortcut”. However, when combined with the Browser Mode, it became significantly more powerful: It enables benchmarking TypeScript code from the command line in a browser environment.

However, integrating these features was initially not possible due to an issue in Vitest. The Vitest team resolved the ticket in a few months, and with a clear communication on the progress, which was great to witness.

Testing And Benchmarking A Sample Project

To demonstrate how easily Vitest and Playwright can be used to test and benchmark TypeScript code using a headless browser, I created a sample project. It aims to compare two methods of drawing thick lines on a canvas element: using CanvasRenderingContext2D.drawRect or using CanvasRenderingContext2D.stroke (with a thick lineWidth).

The project is available here: github.com/jacomyal/canvas-benchmark

The Main Code

Essentially, the two functions I tested and benchmarked look like this:

function fillBasedThickLine(ctx, from, to, thickness, color) {
  const dx = to.x - from.x;
  const dy = to.y - from.y;
  const d = Math.sqrt(dx ** 2 + dy ** 2);
  const angle = Math.atan2(dy, dx);

  ctx.save();
  ctx.fillStyle = color;
  ctx.translate(from.x, from.y);
  ctx.rotate(angle);
  ctx.fillRect(0, 0 - thickness / 2, d, thickness);
  ctx.restore();
}

function strokeBasedThickLine(ctx, from, to, thickness, color) {
  ctx.beginPath();
  ctx.moveTo(from.x, from.y);
  ctx.lineTo(to.x, to.y);

  ctx.lineWidth = thickness;
  ctx.strokeStyle = color;
  ctx.stroke();
}

The exact code is available in the src/index.ts file. To test that they function correctly, I added a small example, where users can click on a canvas to draw thick lines. You can play with it here:

Unit Testing

I began by writing a test for each function to verify that the function colors at least one pixel on a 1x1 canvas. They look like that:

test("it should colorize pixels", () => {
  // Draw a thick line that contains the only pixel
  drawThickLine(ctx, { x: -1, y: -1 }, { x: 1, y: 1 }, 2, "#ff0000");

  // Retrieve the pixel color:
  const {
    data: [r, g, b, a],
  } = ctx.getImageData(0, 0, 1, 1);

  expect([r, g, b, a]).toStrictEqual([255, 0, 0, 255]);
});

The full version with the boilerplate is available in the src/index.spec.ts file.

To run the unit tests, I first installed Vitest and Playwright:

npm install --save-dev vitest @vitest/browser playwright

Playwright also needs to install its headless browsers locally:

npx playwright install

Then, I added a vitest.config.mts file, instructing Vitest to use Playwright and the Browser Mode:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    include: ["*.ts"],
    dir: "src",
    browser: {
      provider: "playwright",
      name: "chromium",
      enabled: true,
      headless: true,
    },
  },
});

At this point, when I ran npx vitest test src/index.spec.ts, it logged the following:

DEV  v1.5.3 canvas-benchmark
Browser runner started at http://localhost:5173/

✓ src/index.spec.ts (2)
✓ fillBasedRectangle (1)
✓ it should colorize pixels
✓ strokeBasedRectangle (1)


✓ it should colorize pixels

Test Files  1 passed (1)
Tests  2 passed (2)
Start at  16:58:52
Duration  800ms (transform 0ms, setup 0ms, collect 20ms, tests 6ms, environment 0ms, prepare 0ms)

Unit tests ran successfully from the command line in a headless browser!

Benchmarking

To benchmark those functions, I wrote something like this:

bench(
  "Canvas methods to draw a thick line",
  () => {
    const angle = 2 * Math.PI * Math.random();
    drawThickLine(
      ctx,
      {
        x: 500 - 500 * Math.cos(angle),
        y: 500 - 500 * Math.sin(angle),
      },
      {
        x: 500 + 500 * Math.cos(angle),
        y: 500 + 500 * Math.sin(angle),
      },
      50,
      getRandomColor(),
    );
  },
  { iterations: 1000 },
);

Again, the full version with the boilerplate is available in the src/index.bench.ts file.

To enable the Benchmarking feature, I added one line to the Vitest config:

export default defineConfig({
  mode: "benchmark",
  test: {
    /* ... */
  },
});

Now, running npx vitest bench ./src/index.bench.ts logs the following output:

DEV  v1.5.3 canvas-benchmark
Browser runner started at http://localhost:5173/

✓ src/index.bench.ts (2) 8624ms
✓ Canvas methods to draw a thick line (2) 8615ms
name                             hz     min       max    mean     p75     p99    p995    p999       rme  samples
· Using a filled rectangle  15,341.30  0.0000  1,785.50  0.0652  0.0000  0.1000  0.1000  0.1000  ±184.32%    29127   fastest
· Using a wide stroke       14,882.96  0.0000  2,427.90  0.0672  0.0000  0.1000  0.1000  0.1000  ±189.11%    37450

BENCH  Summary

Using a filled rectangle - src/index.bench.ts > Canvas methods to draw a thick line
1.03x faster than Using a wide stroke

Both methods exhibit very similar performance. And this insight is automatisable, and reproducible in the command line.

Conclusion

I’ve been developing JavaScript libraries targeting the browser for over 10 years, and proper unit testing has always been challenging. Benchmarking was merely a dream. Now, thanks to Vitest and Playwright, integrating these processes into any TypeScript project has become remarkably easy.