The presence of HTTP request realizations makes frontend testing frameworks like Cypress and Playwright suitable tools for API testing as well.

I was very skeptical about testing API with all-in-one frontend testing tools like Cypress and Playwright, but I decided to give it a try. Jest — another popular frontend testing framework — also can be used for API testing, but it should be extended with third-party libraries like Axios, GOT, node-fetch, etc., to provide HTTP requests.

I have been using Jest + HTTP library in several of my projects, and it has proven to be a simple and reliable tool. It can serve as a starting point for comparison.

Jest

  • Lightweight. Easy to run locally and in CI/CD pipelines.
  • It is a tool for «constructor-based» test infrastructure — you take Jest as a test runner and choose any suitable tool for HTTP requests. It is very convenient if your tests are located inside the tested application codebase and have to depend on the application’s stack: it is already using a specific HTTP library or/and the application already has unit tests on Jest.
  • Manual customization. All «helpers» (getting testing environment settings, bindings for API calls, retries and timeouts, etc.) had to be written, cause nothing goes out of the box.
  • Default reporter well suited for API tests output.

Jest API tests reports

Jest API tests reports

Fig. 1 & 2. Jest API tests reports: pass and fail

Further reading:

Cypress

  • Debugging through UI. Cypress is still a frontend testing tool, so when you wright and debug tests you have to use its GUI for E2E testing.

But there is a very convenient feature in this: you can open DevTools Console in GUI and see the full response object of the request. With a special plugin, you can get the response in a Postman-like view in the main window. In Jest and Playwright, you will have to debug with console.log() for that.

Cypress can check only the status, body, headers, and duration of the response

Fig. 3. Take note of the object in the console: Cypress can check only the status, body, headers, and duration of the response

You can even record videos of how your API tests run and take screenshots in case of failure, but for some skilled automation engineer it would be overhead.

Screenshot of a failed API test

Fig. 4. Screenshot of a failed API test

  • Ambiguous CLI reporter. It looks beautiful, but contains a lot of redundant information. It has a gorgeous stack trace if the test fails at the moment of the request, but it does not help much if fails one of the many checks.

Cypress API tests reports

Cypress API tests reports

Cypress API tests reports

Fig. 5–7. Cypress API tests reports: pass and fail examples

  • Complexity of Cypress test infrastructure. Cypress wraps your tests into its own global variable, and it is daunting to add/use something different (like a third-party library for some precondition/postcondition reasons). Adding an additional package of Cypress into an existing application may also not be painless. For example, it conflicts with Jest, Karma, and Jasmine and needs an individual tsconfig file. The need for additional binary installation may be problematic in CI/CD pipelines.
  • Nice argument failOnStatusCode in the request method.
  • Extremely slow (see «Test execution time» chapter at the end of the article). It takes time to first start Cypress app (a few seconds, even the headless one in CLI) and only then do a test run.

Further reading:

Playwright

  • Lightweight. API testing in Playwright does not require the installation of browsers, you can use it only as a runner (with a built-in HTTP library).
  • All Playwright features are applied to API tests! Retries, parallelism, flexible timeouts, global setup, precise options for each test, etc. — a magnificent set of features, especially if you are already used to spartan Jest test runner.
  • Setting the project through Playwright config allows making requests short and clear: base URL and extra HTTP headers will be hidden from requests in test files.
  • Implicit behavior: Playwright automatically adds storageState into requests inside the context — sometimes it can be useful, but other times it causes a headache.
  • Nice option failOnStatusCode in each request method.
  • Embarrassing output on the default CLI reporter. Anyway, the HTML report at the end of the run can give all the clues.

Playwright API tests reports

Playwright API tests reports

Playwright API tests reports

Fig. 8–10. Playwright API tests reports: pass and fail

Further reading:

Test Execution Time

To compare frameworks by these criteria, I implemented a bunch of the same API tests for each of them (I understand that it may be too simple, but it gives a minimal representation of a tool):

  • 4 GET requests of OpenWeatherMap API;
  • Each test has checks for the status code, one of the headers, and body values;
  • I ran each test suite for each framework 10 times and calculated the average value of test suite execution.

Results of the test suite:

  • Cypress:run command time: 12,96 sec
  • Playwright command time: 3,21 sec
  • Jest command time: 3,84 sec

Jest and Playwright have approximately the same speed (although Playwright is insignificantly faster), but Cypress is 4 times slower! Unfortunately, it is unacceptable for API testing, where tests’ execution time is crucial.

Further reading:

As I mentioned earlier, I was very skeptical about Playwright and Cypress as API testing tools before a try, which is why I was very surprised by the convenience of both of them. If I need to choose a tool for testing API and UI for a new web project, I will definitely take Playwright.

Copy @ Medium