Break-glass environment changes

There are two main reasons that you will want to use a break-glass procedure and bypass Coherence to modify your application.

• Your production environment is down due to a bad environment variable configuration
• Coherence itself (or github) is down, and you need to deploy a hotfix to your production environment

The procedures below are not any different for environments other than production, but the use case is not as common.

For bad environment variable configuration, a fix requires you to edit the variable and then wait for a new pipeline to run in Coherence. Depending on how long your application takes to build, and the cloud provider you are using, this may introduce unacceptable delays for you. If so, there are options available.

Rapidy Fixing a Bad Configuration in Production

AWS

These can be done either in the Cloud Console or with the AWS CLI, as you prefer.

• Create new revision of the task definition by finding the most recent revision of the relevant definition and creating a new revision based on it
• Edit the necessary environment variables in the process. These can be either:
  • Secret references (in which case you will need to edit the secrets directly - you can find these links from the Coherence UI or directly in the AWS Console)
  • Coherence-managed variables such as COHERENCE_ENVIRONMENT_NAME in which case you can edit the values directly in the task definition UI
• Update the ECS Service to deploy the new revision of the task definition, by choosing to update the service's task definition and selecting the revision you created
• The ECS control plane will spin down the old one and spin up the new one, no new container needed build or deployment needed

You'll also need to consider updating scheduled tasks and workers, which have their own configuration for definition. Generally, the best procedure is to update the online services where uptime is critical, while triggering a new pipeline from Coherence, and in doing so both resolving your incident and getting the solution managed by Coherence, both as quickly as possible.

GCP

• Find the relevant Cloud Run service for your application, and select the Revisions tab.
• Create a new revision, making the modifications needed to environment varialbes and secret references. Secrets in GCP for Coherence Variables can be found by using the Manage link in the ... menu on each item in the Coherence UI.
• Save the new revision.
• Set the revision to recieve 100% of traffic for the service in the Cloud Run UI.
  • Cloud Run will deploy the new revision and your application will be back online.

As with AWS above, you'll also need to consider updating scheduled tasks and workers, which have their own configuration for definition. Generally, the best procedure is to update the online services where uptime is critical, while triggering a new pipeline from Coherence, and in doing so both resolving your incident and getting the solution managed by Coherence, both as quickly as possible.

Deploying When Coherence is Down

In many cases, this may be more difficult than the above process. For example if you require database migration or seeding, new infrastructure configuration, or the use of a toolbox for manual migration steps. However, in every case it is possible to sidestep Coherence and deploy directly. We will cover some steps for the most basic scenario below.

This section is a major work in progress - and please remember that we are here to support you & your team over slack/email/phone/video chat and keep your app operational, regardless of our own service status.

• Building & pushing a new container image
  • This can be done locally or using a hosted cloud system such as GCP Cloud Shell
  • docker build against your repo, and then docker push to your repositry, with the new tag, are the important steps
• Creating a new revision
  • See the steps above for AWS & GCP. In this case you are changing the image rather than the environment (or possibly both, depending on the requirements of your change)
• Deploying the new revision
  • See the steps above for updating the Cloud Run or ECS service, they will be the same in this case.

You'll also need to consider updating scheduled tasks and workers, which can be done in a similar manner, but depending on the couping of application versions may or may not be required in your case. The next Coherence deployment will redeploy all the services and components of your application.