How To

Run automatic integration tests against each deployment to a non-production environment

General Information

Add a configuration section to your coherence.yml that looks like:

integration_test:
  type: integration_test
  command: [“npx”, 'cypress', 'run']
  image: 'cypress/included:10.9.1'

Modify the command: and image: to match the requirements defined by your own test scripts.

COHERENCE_BASE_URL will be set as an environment variable that describes the url of the Coherence environment you are running in. You can make requests to this url from within your tests.

Any environment configuration variables that your tests need can be set in the Coherence dashboard with no service attached, and will be injected into the integration_test step in your builds

Full Playwright Example

We have a Coherence application with a service called my-next-app. The coherence.yml looks like this:

my-next-app:
  type: backend
  local_packages: ["node_modules"]
  dev: ["npm", "run", "dev"]
  prod: ["npm", "run", "start"]

Assuming we have a file tree like the following in the repo:

.
|-- Dockerfile
|-- src
|   |-- ...src here
|-- e2e
|   |-- README.md
|   |-- globalSetup.ts
|   |-- package.json
|   |-- playwright.config.ts
|   |-- state.json
|   |-- tests
|   |   `-- smoke.spec.ts
|   `-- yarn.lock
|-- index.html
|-- package.json
|-- vite.config.js
`-- yarn.lock

The following yml in the coherence.yml (top-level - see here) will be all we need to run our tests.

integration_test:
  type: integration_test
  command: ["cd", "e2e", "&&", "yarn", "install", "&&", "yarn", "playwright", "test"]
  image: "mcr.microsoft.com/playwright:v1.36.1-focal"

This will result in each Coherence pipeline to a branch environment running the integration tests after deploy. The smoke.spec.ts script uses the COHERENCE_BASE_URL to know what URL to hit in each environment by reading it from the environment. You can do this using baseURL: process.env.COHERENCE_BASE_URL in the use block of your playwright.config.ts. Using reporter: 'list' outputs failure info to the terminal (instead of a local html server), this should also be set. You might also want to replace the process.env.CI check with the Coherence-provided COHERENCE_CI var like process.env.COHERENCE_CI in the plawright.config.ts file.

In a Workspace, the COHERENCE_BASE_URL will be set to the appropriate local port/path for the app. In a preview environment, the COHERENCE_BASE_URL will be set to the environment's URL.

To get trace info out of your Workspaces (since we won't forward the port for the debug web server), one flow is to do yarn playwright test --trace on. After the test completes:

  • Refresh your file browser
  • Open the test-results folder and find the test you want
  • Download the trace.zip file by right clicking it and selecting download
  • Go to test.playwright.dev and drag the zip folder onto the screen
  • To specify where to save the traces, add outputDir: 'test-results/' to your plawright.config.ts file
  • You can use reporter: 'list' to disable the default debug web server starting after the test run

Here's a complete updated coherence.yml for the app as outlined:

backend:
  dev:
    - npm
    - run
    - dev
  prod:
    - npm
    - run
    - start
  type: backend
  local_packages:
    - node_modules
integration_test:
  type: integration_test
  image: mcr.microsoft.com/playwright:v1.36.1-focal
  command:
    - cd
    - e2e
    - '&&'
    - yarn
    - install
    - '&&'
    - yarn
    - playwright
    - test

And here's the playwright.config.ts:

import { defineConfig, devices } from '@playwright/test';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
// require('dotenv').config();

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
  testDir: './tests',
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.COHERENCE_CI,
  /* Retry on CI only */
  retries: process.env.COHERENCE_CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.COHERENCE_CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: 'list',
  /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
  use: {
    /* Base URL to use in actions like `await page.goto('/')`. */
    baseURL: process.env.COHERENCE_BASE_URL,

    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on-first-retry',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },

    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },

    /* Test against mobile viewports. */
    // {
    //   name: 'Mobile Chrome',
    //   use: { ...devices['Pixel 5'] },
    // },
    // {
    //   name: 'Mobile Safari',
    //   use: { ...devices['iPhone 12'] },
    // },

    /* Test against branded browsers. */
    // {
    //   name: 'Microsoft Edge',
    //   use: { ...devices['Desktop Edge'], channel: 'msedge' },
    // },
    // {
    //   name: 'Google Chrome',
    //   use: { ...devices['Desktop Chrome'], channel: 'chrome' },
    // },
  ],

  /* Run your local dev server before starting the tests */
  // webServer: {
  //   command: 'npm run start',
  //   url: 'http://127.0.0.1:3000',
  //   reuseExistingServer: !process.env.CI,
  // },

  outputDir: 'test-results/'
});

An Advanced Note

Generally, you can ignore the info below, discuss with us if any concerns here, but just to clarify the behavior...

Why the yarn install step in the command above? The package.json for my app will be installed by my own Dockerfile in this case, but since the recommended pattern for integration tests is to use a 3rd-party image, that doesn't help us for the integration test step where we are using the playwright image from mcr below.

  • These project-provided images work well because they have all browser OS and package dependencies properly installed to be used as expected by playwright. Similar images are available for other popular testing tools like Cypress.

In order to install our deps in the playwright container we are running in, we run the yarn install step as part of our integration test command in the coherence.yml. This is slow and might have occasional network errors, so is not ideal. We could avoid that by building our own image based on that with the packages installed at build time and using it here (see the dockerhub integration if relevant). Get in touch if we can help!

Previous
Integrate GitHub Actions CI/CD pipelines
Next
Add Custom Cloud Resources to Environments