Cypress on AWS Lambda: part 2

Our goal is to sabotage the ever-increasing slowness of end-to-end tests.

(If we don't, they'll probably keep getting slower and slower until we gleefully delete them. And then a month later, we'll add them back after a production deploy goes sideways and we realize that it could have been prevented by the tests. Ad infinitum.)

In part one, we established a mechanism for running a single Cypress spec file on AWS Lambda. Now we will employ that mechanism to run all the spec files in the cypress-example-kitchensink project in parallel.

Approach

First, I forked cypress-example-kitchensink and deployed the app under test to Netlify, which took about 30 seconds (thanks Netlify). The deploy is here.

The tests in cypress-example-kitchensink assume that the app is being served locally; I changed the tests to point at the Netlify deploy, and copied them into cypress-lambda (the project from part 1), so they'll be deployed as part of our Lambda function package.

Back in cypress-lambda, we'll ask terraform to write our deployed Lambda function's ARN to a file, so we can get a handle to it and invoke it:

resource "local_file" "lambda_arn" {
  content  = "${aws_lambda_function.cypress_runner.arn}"
  filename = "deployed_lambda_arn"
}

Now, after we do terraform apply, the file exists:

$ cat deployed_lambda_arn
arn:aws:lambda:us-west-2:123456789123:function:cypress_runner

Next we add a Node script that we'll run locally, index.js:

const fs = require("fs");
const AWS = require("aws-sdk");
const glob = require("glob");

const lambda = new AWS.Lambda({ region: "us-west-2" });

const lambdaArn = fs.readFileSync("./deployed_lambda_arn").toString();

async function main() {
  const files = glob
    .sync("cypress/integration/**/*.spec.js", {
      cwd: "lambda"
    })
    .map(file => `/tmp/${file}`);

  try {
    const results = await Promise.all(
      files.map(file => {
        return lambda
          .invoke({
            FunctionName: lambdaArn,
            Payload: JSON.stringify({ cypressSpec: file })
          })
          .promise();
      })
    );

    results.forEach((result, idx) => {
      fs.writeFileSync(
        `reports/mochawesome-${idx}.json`,
        JSON.parse(result.Payload)
      );
    });
  } catch (e) {
    console.error(e);
  }
}

main();

This script:

  • References the Lambda function ARN from the file written by terraform
  • Makes a list of all the individual spec file paths
  • For each spec file, concurrently invokes the Lambda function, passing in just that spec file
  • Writes each Lambda function's response payload to a file in reports/

We've tweaked the Lambda function handler so that when Cypress runs, it'll use mochawesome as a reporter and return the report to the caller (i.e., our script). Then, we can merge the results from the different function invocations, and generate an HTML report.

We'll add shortcuts for merging and generating reports to package.json:

  "scripts": {
    "test": "npm run clean-reports && node index.js && npm run merge-reports && npm run generate-html-report",
    "merge-reports": "npx mochawesome-merge --reportDir reports > mochawesome.json",
    "generate-html-report": "npx mochawesome-report-generator mochawesome.json",
    "clean-reports": "rm -rf reports && mkdir reports"
  },

So now when we do npm test, it will run and create a report.

Results

As expected, running all 18 spec files (which is 92 specs total) in parallel saves us quite a bit of time:

Description Time
Run all spec files sequentially from my laptop 5m
Run all spec files sequentially in one Lambda function invocation 1m30s
Run all spec files in parallel on Lambda (cold) 36s
Run all spec files in parallel on Lambda (warm) 18s

After running the tests via npm test, we get a stylish report:

Report courtesy of mochawesome

Conclusions

Unless there is some compelling reason it's not possible, I think maximizing parallelization of end-to-end tests using functions-as-a-service is a pretty good idea.

Here we were able to get the tests running in a fraction of their original run time, and the relative gains can be much larger as the number of spec files increases. For example, the folks at Blackboard parallelized their Selenium e2e tests via Lambda and found:

One year ago, one of our UI test suites took hours to run. Last month, it took 16 minutes. Today, it takes 39 seconds.

There are several limitations of the current approach. It is a prototype that hasn't yet been battle-tested, and I wouldn't expect the simplified approach in index.js to scale to a number of spec files in the thousands. The story around error handling is...weak. And there was an issue in one of the kitchensink project's 19 spec files (actions.spec.js) that was causing the Chromium renderer process to crash in the Lambda execution environment, which I didn't dig into, and instead commented out that file :p

Even with those limitations, it still represents huge potential time-savings.

Check out the repo to deploy this in your AWS account with your own Cypress end-to-end tests, and let me know how it goes.

And stay tuned for part 3, where we'll plug this into a CI/CD pipeline so that it's fully automated, running against every PR.

Show Comments