Menu
Tags

Cypress vs. Playwright - Basic Testing

Published on Nov 20, 2023

#cypress#playwright#automation

Table of Contents

Battle of the Browser Automation Tools

I’ve been using Cypress at work for the last 3 years, and I’ve loved it. I’ve felt that the developer experience is about as good as it gets, and the GUI tool makes it easy to toss into the hands of developers and get them writing tests during feature development. (As much as I’ve dreamed about getting the team to move to test driven development, it never came to be.) However, recently Cypress has made some not so great business decisions that has made me fear for its future a bit. Building an open source tool like Cypress, then limiting what users can do with it usually doesn’t go over well, and I’m expecting a significant migration over to Playwright. So I’ve wanted to play with Playwright, but with out test suite already built and mature at work, I haven’t had time to do so at work yet. So I’ll do it here. Over a few posts, I’m going to be writing lots of tests in Cypress and Playwright and doing my best to draw some conclusions.

In todays post, we’ll look at the initial configuration and writing of some basic tests.

I’m well aware of my bias for Cypress with the amount of times I’ve committed to it. I’m doing my best to put myself in the shoes of someone fresh to the tool, but I expect I can’t scrub all my bias away.

Initial Config

Getting started with a new testing framework can be tedious when your dropping it into an existing project. So, lets compare the process of setting up both Cypress and Playwright.

Cypress

Cypress seems to have invested a good deal into this initial setup work. The process is as simple as it could get. First, we install the Cypress package, and then we run it.

pnpm install -D cypress
pnpx cypress open

The Cypress GUI will open up and take care of the work of spitting out example tests, fixtures, config files, etc. If we poke through the GUI, we can create and run new test from the GUI itself. It is all as plesant as could be desired.

[Placeholder - Add Video Animation of Install and initial run here]

…however, on my project I hit my first bug upon running my first test.

TS5053: Option 'sourceMap' cannot be specified with option 'inlineSourceMap'.

Assuming there is something it doesn’t like in my .tsconfig, and with my past experience with Cypress, I’m going to quickly assume I’m going to need a Cypress specific tsconfig.json within the /cypress directory to over-ride any of the root config it doesn’t like. I’ll make a new .tsconfig and drop in Cypress’ recommended config first.

cypress/tsconfig.json
{
  "compilerOptions": {
    "target": "es5",
    "lib": ["es5", "dom"],
    "types": ["cypress", "node"]
  },
  "include": ["**/*.ts"]
}

…a bit of Googling reveals that I might just have to disabled source mapping in this new config.

cypress/tsconfig.json
{
    "compilerOptions": {
        ...
        ++"sourceMap": false
        ...
    }
}

And we are all set. We are off and running tests with no issues.

Playwright

I’m building this blog in Sveltekit, and it allowed me to initialize the project with Playwright pre-configured…but I’m wanted to experience it from the start, so I stripped it all out and followed the docs from the start.

pnpm create playwright

Their little CLI installer seems pretty strightforward. Where do you want tests stored? Do you want to install browsers? etc. Following their getting started guide printed after install, I’m able to run the tests in their GUI.

pnpm exec playwright test --ui

And we are off to running tests.

Configuring global parameters

I want to define some common variables that all tests will need. I’m intimately aware of Cypress’ use of their .config.ts files. I’ve used this plenty to create different config files for each region of our environment. Playwright, on the other hand is completely new, though their docs seem pretty clear to me with the use of ‘projects’ array in their config to represent different regions, as well as browsers and viewport sizes. However, in the case of running these two side-by-side, I want a common place to store values for both. I’m thinking the right route is using a .env and importing them into each config file.

Cypress

With Cypress, importing the env file itself can be tricky. With their Webpack config, loading the .env file in the test file itself is a no go. But we can import it into the config file, and spread the results into the env section of the config, and then access them from the tests using their Cypress.env(‘value’) function.

cypress.config.ts
import { defineConfig } from "cypress";
import dotenv from 'dotenv';

dotenv.config({path: 'public.env.test'});
const env = process.env;

export default defineConfig({
  reporter: 'junit', 
  reporterOptions: {
    mochaFile: './cypress/results/junit/results.xml'
  },
  env: {
    'baseURL': 'http://localhost:5173',
    ...env
  },
  e2e: {
    setupNodeEvents(on, config) {
      // implement node event listeners here
    },
  },
});

Playwright

I was quickly able to find Playwright Parameterization doc, but this ends up being a bit heavy handed for what I want to do with the defining of projects. I can see the power of this when it comes to larger projects with multiple environments, but for our basic site, we just need one simple project.

I found it more straightfoward to simply import the .env file in the test itself, and access the params with the process.env.value syntax.

Basic initial testing set

I wanted to spin up a few quick tests just to start running and getting things moving. I wrote a test with Cypress, then re-wrote it for Playright, then rinsed and repeated. Shoutout to Github Copilot, because it knocked these out in no time at all. It seemed to pick up exactly what I was doing, and quickly recreated the tests.

Cypress Test Code

cypress/e2e/test.cy.ts
describe('template spec', () => {
  beforeEach(()=>{
    cy.visit(Cypress.env('baseURL'));
  })

  it('I can access the site', () => {
    cy.visit(Cypress.env('baseURL'));
    cy.get('[data-testid="title"]').should('contain', Cypress.env('title'))
  })
  
  it('I can read the first blog post', () => {
    cy.visit(Cypress.env('baseURL'));
    cy.get('[data-testid="blog-card-title"]').contains(Cypress.env('testPostTitle')).click();
    cy.get('[data-testid="blog-post-title"]').should('contain', Cypress.env('testPostTitle'));
  })

  it('I can see table of contents on large viewports', () => {
    cy.visit(Cypress.env('baseURL'));
    cy.get('[data-testid="blog-card-title"]').contains(Cypress.env('testPostTitle')).click();
    cy.viewport(1280, 720);
    cy.wait(1000);
    cy.get('[data-testid="toc"]').should('be.visible');
  })

  it('I cannot see table of contents on small viewports', () => {
    cy.visit(Cypress.env('baseURL'));
    cy.get('[data-testid="blog-card-title"]').contains(Cypress.env('testPostTitle')).click();
    cy.viewport(320, 480);
    cy.get('[data-testid="toc"]').should('not.exist');
  })
})

Playwright Test Code

playwright/e2e/test.spec.ts
import { test, expect } from '@playwright/test';
import dotenv from 'dotenv';

dotenv.config({path: '.env.test'});

test('I can acess the site', async ({ page }) => {
  await page.goto(process.env.baseURL);
  await expect(page.getByTestId('title')).toHaveText(process.env.title);
});

test('I can read the first blog post', async ({ page }) => {
  await page.goto(process.env.baseURL);
  await page.getByText(process.env.testPostTitle).first().click();
  await expect(page.getByTestId('blog-post-title')).toHaveText(process.env.testPostTitle);
});

test('I can see table of contents on large viewports', async({ page }) => {
  await page.goto(process.env.baseURL);
  await page.getByText(process.env.testPostTitle).click();
  await page.setViewportSize({ width: 2000, height: 720 });
  await page.waitForTimeout(1000);
  await expect(page.getByTestId('toc')).toBeVisible();
});

test('I cannot see table of contents on small viewports', async({ page }) => {
  await page.goto(process.env.baseURL);
  await page.getByText(process.env.testPostTitle).click();
  await page.setViewportSize({ width: 320, height: 480 });
  await expect(page.getByTestId('toc')).not.toBeVisible();
});

Running the tests

Getting these tests ready to run quickly and easily is huge. I spun up a few new scripts in the package.json so we can run the tests both in GUIs as well as headless, and then let em’ run. Both ran easily enough, but of course ran into some issues. I’ll say that Playwright didn’t give me any feedback in the GUI when there was a minor syntax issue in the test code, but Cypress made it nice and clear with a line number to go checkout. with Playwright, I have to click on the ‘source’ tab in the GUI to see the error.

After running both test sets, I got a consistant result between the two, with one test failing, revealing an issue with my blog code I didn’t realize before, exactly what automated testing is for…resolving that problem will be for a later date, and fankly, having a failing test is kind of a nice thing to have while comparing these two.

The Outputs

I set both tools to output a Junit report, I was curious to see the difference between the two outputs. The reports looked similar, with the main difference being how they reported the error. Let’s take a look:

Cypress Failure Note

cypress/results/junit/results.xml
<failure message="Timed out retrying after 4000ms: expected &apos;&lt;div#side-toc.logo.s-KnlqTvvrWAx4&gt;&apos; to be &apos;visible&apos;

This element `&lt;div#side-toc.logo.s-KnlqTvvrWAx4&gt;` is not visible because it has CSS property: `display: none`" type="AssertionError"><![CDATA[AssertionError: Timed out retrying after 4000ms: expected '<div#side-toc.logo.s-KnlqTvvrWAx4>' to be 'visible'

This element `<div#side-toc.logo.s-KnlqTvvrWAx4>` is not visible because it has CSS property: `display: none`
    at Context.eval (webpack://personal-2023/./cypress/e2e/spec.cy.ts:19:0)]]></failure>

Playwright Failure Note

playwright/results/junit/results.xml
<failure message="example.spec.ts:17:1 I can see table of contents on large viewports" type="FAILURE">
<![CDATA[  [chromium] › example.spec.ts:17:1 › I can see table of contents on large viewports ───────────────

    Error: Timed out 5000ms waiting for expect(locator).toBeVisible()

    Locator: getByTestId('toc')
    Expected: visible
    Received: hidden
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for getByTestId('toc')


      20 |   await page.setViewportSize({ width: 2000, height: 720 });
      21 |   await page.waitForTimeout(1000);
    > 22 |   await expect(page.getByTestId('toc')).toBeVisible();
         |                                         ^
      23 | });
      24 |
      25 | test('I cannot see table of contents on small viewports', async({ page }) => {

        at /home/barnes/dev/personal-blog/playwright/e2e/example.spec.ts:22:41
]]>
</failure>

Both of these are effective and clear are reporting the problem, but personally, I feel like Cypress has a bit of an easier read out. In particular, I think of the case of non-technical team members reading these reports, the Cypress report is likely much more for them to go on. However, realistically, it would be our technical team members who hit these errors the most frequently, so its a bit of a wash.

Wrapping up:

Right now, nothing stood out as a game changer with Playwright compared to my experience with Cypress. Plenty of minor hiccups, but ultimately, I had no issues getting everything up and running. The next step is get these tests running on CI/CD and see what ends up being more effective there. In my next post, I’ll outline that process.