Table of Contents

• Defining the deployment flow
• Handling Secrets
• Test Reporting
  • Cypress
  • Playwright
• The Github Workflow
• Displaying the results
• Wrapping Up

Defining the deployment flow

I’m using Vercel to host this blog, and hosting the repo over on Github. I’m a day to day Bitbucket & AWS user at my place of work, and frankly it has left a bad taste in my mouth when it comes to CI/CD, so I’m looking forward to using this opportunity to use Github Actions, which I hope will be a much cleaner tool to use.

First, lets outline the flow I’m imaging:

1. Install all deps needed.
2. Run the Sveltekit Build
3. Start the preview server.
4. Run Cypress Tests
5. Run Playwright Tests
6. Store the test results in an S3 bucket, or some kind of artifact bucket within Github.
7. Deploy to Vercel

Finally, I want to display the results here, so I’ll need to spin up some components and a new route to take in the S3 bucket contents and have them rendered on this front page. A bit unusual for a blog to publish their test results, but this blog is about testing, so why not.

Handling Secrets

I have my common .env.test file I’m using to import common parameters for tests, but that runs into the issue of not being commited to the repo, so just have to redefine this file to make sure it makes it into the repo.

Secrets will be handled with Github’s built into repo secrets.

Test Reporting

I want to have as close to perfect testing parity between Cypress and Playwright, making it easy to compare the results themselves. I’ve always been fond of the auto-generated Mochawesome HTML reports, something nice and easy to share with the larger team, easy to read and has all the details needed. So lets configure the reporters.

Cypress

We’ll use the cypress-mochawesome-reporter NPM package to generate the HTML report:

pnpm install -D cypress-mochawesome-reporter

And then configure it in the cypress config.

...
export default defineConfig({
  reporter: "cypress-mochawesome-reporter",
  reporterOptions: {
    "reportDir": "cypress/results",
    "charts": true,
    "embedScreenshots": true,
    "inlineAssets": true,
    "overwrite": false,
    "html": false,
    "json": true,
    "saveJson": true,
  },
  ...
});

I’m using inline assets and embed screenshots here to make sure I have a simple flat HTML file that I can load from the S3 bucket later on.

I’ll end up with a results directory with screenshots, HTML and JSON that I’ll upload to an S3 bucket upon test completion.

Playwright

I’ll stick with Playwrights built in reporter as it seems to do everything I’d like. Configuration is certainly more straightforward with Playwright.

...
reporter: [['json', { outputFile: './playwright/results/json/results.json' }], ['html', {outputFolder: './playwright/results/html', open: 'never' }]],
...

The Github Workflow

I built this workflow based off of Vercel’s documentation, and spliced in my tests and S3 upload steps. Nothing here is too crazy. The only unique thing is the collection of the timestamp that I am using as a directory label in S3.

You’ll note that I have the testing steps configured to not fail any builds. At this point, I want to record and publish all test results, pass or fail. Down the line I may strip this out.

name: Vercel Preview Deployment
env:
  VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
  VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
  AWS_REGION: ${{ secrets.AWS_REGION }}
  AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}
on:
  push:
    branches-ignore:
      - main
jobs:
  Deploy-Preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: "Set current date as env variable"
        run: |
          echo "builddate=$(date +'%Y-%m-%d:%H:$M:%S')" >> $GITHUB_OUTPUT
        id: version
      - name: Clear lockfile
        run: rm -rf pnpm-lock.yaml
      - name: Install Vercel CLI
        run: npm install --global vercel@latest
      - name: Pull Vercel Environment Information
        run: vercel pull --yes --environment=preview --token=${{ secrets.VERCEL_TOKEN }}
      - name: Install Deps
        run: npm install
      - name: Build App
        run: npm run build
      - name: Install Playwright
        run: npx playwright install
      - name: Install Cypress
        run: npx cypress install
      - name: Start preview server
        run: npm run preview &
      - name: Run Cypress Tests
        continue-on-error: true
        run: npm run cypress
      - name: Run Playwright Tests
        continue-on-error: true
        run: npm run playwright
      - name: Store Results in AWS S3
        run: |
          aws s3 sync ./cypress/results/ s3://blog-test-results/${{ steps.version.outputs.builddate }}/cypress
          aws s3 sync ./playwright/results/ s3://blog-test-results/${{ steps.version.outputs.builddate }}/playwright
      - name: Deploy Project Artifacts to Vercel
        run: vercel deploy --prebuilt --token=${{ secrets.VERCEL_TOKEN }}

Displaying the results

Testing is only as good as what is done with test results. While I could let Playwright and Cypress dump their results to the build terminal and be done with it, I wanted more, and something I could share. So lets walk through what we’ve got:

1. On runs, we store all test results in a directory with a timestamp prefix on a remote S3 bucket.
2. We build our application with this timestamp, which it’ll use to go and get the data from the S3 bucket to rendered on a new ‘test-results’ route.

Right now, I took the easy path and place the HTML reports inside of iFrames, and set my bucket to be completely open so I don’t have to fuss too much with pre-signed URLs and all that jazz. Down the line I plan to ingest the test results JSON and create my own output display that is something that could be re-used wherever needed. A component that takes in the test result JSON and displays it cleanly.

For now, the route is pretty straight-forward. The +page.server.ts file will collect all S3 items from the given timestamp prefix and pass the presigned URLs to the front end.

import type { PageServerLoad } from './$types';
import { S3Client, ListObjectsCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import  { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { TIMESTAMP, AWS_REGION, AWS_BUCKET_NAME, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY } from '$env/static/private';

const s3client = new S3Client({
	region: AWS_REGION,
	credentials: {
		accessKeyId: AWS_ACCESS_KEY_ID,
		secretAccessKey: AWS_SECRET_ACCESS_KEY
	}
});

const getPresignedUrls = async (prefix: string) => {
  const command = new ListObjectsCommand({
	  Bucket: AWS_BUCKET_NAME, 
	  Prefix: prefix
	});
	const response = await s3client.send(command);
	const urls = response.Contents!.map(async item => {
	  const command = new GetObjectCommand({
		  Bucket: AWS_BUCKET_NAME,
		  Key: item.Key!
	  });
	  let url = await getSignedUrl(s3client, command, { expiresIn: 3600 });
	  return url;
	});
	const resolvedUrls = await Promise.all(urls);
	return resolvedUrls;
}

export const load = (async () => {
	const urls = await getPresignedUrls(TIMESTAMP);
	return { urls, TIMESTAMP }
}) satisfies PageServerLoad;

and here is the front-end of the page:

<script lang="ts">export let data;
</script>

<h1>Results from {data.TIMESTAMP}</h1>

{#each data.urls as url}
    {#if url.includes('cypress/index.html')}
        <h2><a href={url} target="_blank">Cypress</a></h2>
        <iframe title="Cypress Test Results" class="frame" src={url}></iframe>
    {/if}
    {#if url.includes('playwright/html/index.html')}
        <h2><a href={url} target="_blank">Playwright</a></h2>
        <iframe title="Playwright Test Results" class="frame" src={url}></iframe>

    {/if}
{/each}

<style>
    .frame {
        width: 100%;
        height: 75vh;
        border: 10px solid black;
        margin: 2rem 0;
    }
</style>

Wrapping Up

We’ve got a pretty solid deployment flow, and test results being rendered out on our site now. Things are looking pretty great if you ask me. Next steps will be some general clean up and build optimizations to speed up our overall build and testing times. Then we’ll build some components to render the test results natively in our blog, with the intent of creating some universal web components to ingest the test result JSON and display anywhere needed.