Services

A Coherence application is composed of one or more services. Each service has a type (frontend or backend), creates its own deployment, and defines its own containers, resources (e.g. database, object storage, or cache), and workers/crons.

Service Types

You can have many instances of each service type, but the name and url_path of each one must be unique. The name of a service is defined by the top-level key in the yml.

• backend
• frontend

Common config

For all service types, these attributes are used.

• repo_path tells Coherence where in the repo the service is located. It will be used as the docker build context for the service's container, and as the relative base for the assets_path to discover frontend assets after they are compiled.
  • This value is relative to your repo's root.
• url_path will configure the load balancer for the application to send traffic with this path prefix to the service. Each environment on Coherence get's its own domain, so the path is purely about service routing within an environment, not across environments. The default is /.

Build glob configuration

By default, if a repo_path is configured for a service, when Coherence receives a github webhook for a push (including a merge), it will generate a diff of the files changed in that commit and the last successful pipeline. If none of the files match repo_path/**/*, the service will be ignored for the pipeline generated by the webhook.

Additionally, you can specify build_config object for each service with included_paths and excluded_paths. These are glob patterns that are attached to the repo_path of the service and used with the same diff detection (based on filenames) described above. This can be used to control what services are generated from a push, either with multiple services in one app, or multiple apps in one repo.

For example:

service_1:
  type: frontend
  repo_path: baz
  ...
  build_config:
     # for globs, ** = folder, * = file
     # appended to repo_path, e.g. the path that will trigger a build is baz/foo/* or baz/foo/**/*
    included_paths: ["foo/**/*", "foo/*"]
    excluded_paths: ["README.md"]

Build timeouts (AWS only)

For AWS applications, the following build tasks are considered failed if they have not completed within 20 minutes: test, migrate, seed, snapshot, deploy

If additional time is required for any task it is configurable at the service level in your coherence.yml . The default timeout (20 mins) can also be overwritten to allow more time for all tasks. e.g.

backend:
  build_config:
    timeouts:
      default: 20
      test: 30
      migrate: 20
      snapshot: 40
      seed: 50
      deploy: 30

Dockerfiles

Docker behavior

Each service runs in its own docker container, so you don't need to do anything special to generate a container for each service. By default, each pipeline for a service will build a container using the repo_path as the build context, and the file Dockerfile at the root of the repo_path as the image's definition.

Customizing Dockerfile location

You can use a different filename than Dockerfile for your service. You can also sepcify different Dockerfile's for use in dev (in a Workspace) and in a deployed cloud environment.

To do this, you can specify an attribute dockerfile along with the command under the dev and prod keys for each service. The paths provided are relative to the repo_path which will remain the build context. For example:

dev:
  command: ["run", "command"]
  dockerfile: "Dockerfile.dev"
prod:
  command: ["run", "command"]
  dockerfile: "prod/Dockerfile.prod"

Customizing Dockerfile target

If your dockerfile utilizes multi stage builds and you'd like to target a stage other than the last stage, that can be added to the dev and/or prod commands as a build_target. It is equivalent to adding the --target STAGE argument to the docker build command (see: https://docs.docker.com/build/building/multi-stage/#stop-at-a-specific-build-stage).

dev:
  command: ["run", "command"]
  dockerfile: "Dockerfile.dev"
  build_target: "dev_stage"
prod:
  command: ["run", "command"]
  dockerfile: "prod/Dockerfile.prod"
  build_target: "prod_stage"

Using Nixpacks

If you don't provide a Dockerfile for a service, we will attempt to build a container for your service using Nixpacks. This works well for many common applications!

Backend Services

Please note that your service should look for a PORT environment variable, and accept connections from host 0.0.0.0 on that port (this is the standard expectation of Cloud Run and ECS/Fargate).

dev

Will run in your container on a Workspace.

prod

Will run when serving your container in a review or production environment. See details for AWS and GCP

workers

• The service container will be run as a deployment per worker with the command specified.
• You can manually scale replicas per-environment in the Coherence UI (Environment's infra tab).
• AWS will default autoscale at 70% of CPU use.

On Workspaces:

• The special worker named dev_workspace will replace all workers on a workspace with the given command (to save resources in dev and listen to multiple queues or run multiple jobs).
• If dev_workspace worker is not provided, all workers will be run in a workspace.
• The special worker named preview_environment will replace all workers in a preview environment with the given command (to save resources in dev and listen to multiple queues or run multiple jobs).
• If preview_environment worker is not provided, all workers will be run in each preview environment.

scheduled_tasks

backend:
  scheduled_tasks:
  - name: task 1
    command: [“sleep”]
    schedule: "* * * * *"

See cronjobs#schedule for the schedule element, even on AWS apps. We will normalize the syntax into the ECS configuration if required.

The command provided will be passed to sh -c “command”, so it will support chaining with e.g. && or ;

• On GCP, will share the kubernetes cluster with workers. Coherence will create CronJob resources in the cluster at the schedule specified.
• On AWS, will share the ECS cluster with the HTTP services in your Coherence application. Each service's scheduled tasks will deploy ECS scheduled_task instances with a target of the service. This means that they will use the same container image as the service, albeit with whatever command you specify for the task.

system

backend:
  system:
    cpu: 2
    memory: 2G
    dev:
      cpu: 4
      memory: 3G
    preview:
      cpu: 1
      memory: 1G
    health_check: "/health"
    platform_settings:
      min_scale: 2
      max_scale: 6
      throttle_cpu: true

  workers:
  - name: my_example_worker
    command: ["python", "worker.py"]
    system:
      cpu: 4
      memory: 3G
      preview:
        cpu: 3
        memory: 2G

  scheduled_tasks:
  - name: sample cron task
    command: ["python", "-u", "cron/do_some_things.py"]
    schedule: "*/2 * * * *"
    system:
      cpu: 4
      memory: 3G
      preview:
        cpu: 3
        memory: 2G

• Resources cpu and memory can be configured in the system block, and will be used in dev, review, and production environments for workers, tasks, and backend web processes.
  • dev is also available as a nested value inside system, and cpu and memory can be defined seperately for dev use there. These settings apply to workspaces.
  • preview is available as a nested value inside system, and cpu and memory can be defined seperately for preview use there. These settings apply to all non-static environments.
• Scheduled tasks and workers also accept a system configuration. For these resources, only cpu and memory (incl. preview cpu/mem) are supported as in the example above. If this config isn't present, cpu and memory are inherited from the service.
• health_check: allows you to configure the health check endpoint for your backend service.
• For GCP, min_scale defaults to 0 (which means "cold start" boot times) and you can set it to a higher value to ensure this many replicas are always running.
  • You can also change throttle_cpu to false to keep cpu always-allocated.
  • Keep in mind the cost of these decisions.
  • You can change runtime generation in Cloud Run between gen1 and gen2.
• For AWS, min_scale defaults to 1 and it can be increased to ensure more replicas are always running.
  • You can also optionally set max_scale (default: 4) to control the maximum number of replicas.
  • Autoscaling rules are put in place by default and will manage scaling your service from min_scale to max_scale based on CPU usage (>70%).
  • N.B. Values provided for min_scale and max_scale will only overwrite defaults for production services.

resources

Supported types are database, cache, object_storage, filesystem and message_queue which will be used to provision the appropriate cloud resources (e.g. Memorystore/Cloud SQL/GCS on GCP, Elasticache/RDS/S3/SQS on AWS). Multiple resources of each type can be configured, but their names must be distinct.

Note message_queue and filesystem only available on AWS. If you would like to use a these on GCP, please get in touch.

backend:
  resources:
  - name: db1
    engine: postgres
    version: 13
    type: database
    adapter: optional_string
    snapshot_file_path: "bucket_name/file_name.sql"
  - name: redis
    engine: redis
    version: 4
    type: cache
  - name: test_bucket
    type: object_storage
    cors:
      - allowed_methods: ["GET", "POST", "PUT", "DELETE"]
        allowed_origins: ["www.example.com"]
      - allowed_methods: ["GET"]
        allowed_origins: ["*"]
  - name: test_queue
    type: message_queue
    fifo: True
  - name: vol1
    type: filesystem
    use_existing:
      project_type: review
      instance_name: "YOUR_EFS_FILESYSTEM_ID"

Types

• Backend resources support database, cache, object_storage, filesystem, and message_queue types.
• Under the hood these map to Cloud SQL, Memorystore and Cloud Storage in GCP, or to RDS, Elasticache, S3 and SQS in AWS.
• The engine and version attributes will accept any valid values for those platforms for database and cache types.

Object Storage

• The object storage type takes two attributes: name and a cors block. The cors block consists of pairs of allowed_methods and allowed_origins lists, which will be applied as the Cross-Origin Resource Sharing rules on your bucket. You can add as many CORS pairs to your bucket as needed.
• name is for your use and will be used to create the bucket in your cloud provider, with the addition of your environment name and a unique identifier appended.
• A use_existing block can be configured to add an object storage bucket that is not managed by Coherence that you want to use from your Coherence environments. the project_type attribute can be review or production:

  - name: test_bucket
    type: object_storage
    use_existing:
      project_type: review
      instance_name: EXISTING-BUCKET-NAME

Message Queue (AWS Only)

• The message queue type takes two attributes: name and a fifo boolean. The fifo boolean will control whether your queue is created as an AWS standard queue or first in first out queue.
• name is for your use and will be used to create the queue in your cloud provider, with the addition of your environment name and a unique identifier appended.
• A use_existing block can be configured to add a message queue that is not managed by Coherence that you want to use from your Coherence environments. the project_type attribute can be review or production:

  - name: test_queue
    type: message_queue
    use_existing:
      project_type: review
      queue_name: EXISTING-QUEUE-NAME

EFS Filesystem (AWS Only)

for EFS support, create in AWS yourself (outside Coherence) in the same AWS account you set up your application in during onboarding for the relevant collection (review or production). If provided, EFS will be mounted in the service's ECS tasks, with containerPath set to the name of the resource (e.g. vol1 in the example above).

Database configuration

• The default DATABASE_URL generated for your app's variables will use a lowercased string of the engine name provided as the adapter. (e.g. adapter://username:password@host:port/db_name) You may provide a different value by setting the adapter property of the resource to a string, e.g. postgresql instead of the default postgres is a common choice in python programs.
• For AWS, there is an optional use_db_proxy setting which defaults to true. By default, as a best practice applications will connect via an AWS database proxy whenever possible. If for some reason this will not work for your application (e.g. when using RLS), it can be disabled by setting use_db_proxy to false:

- name: db1
  engine: postgres
  version: 13
  type: database
  use_db_proxy: false

Loading data into databases

• The migration and seed commands for the service will run whatever command you specify inside the service's container, both during Workspace start-up as well as in a CI/CD pipeline for each environment. It's up to you to make your seed command idempotent.
• Additionally, you can provide a snapshot_file_path attribute for each database resource. This can be for example a dump of your staging environment. The snapshot file will be automatically downloaded and loaded into the database for a new Workspace or non-production environment (either)
  • The snapshot_file_path should be the bucket_name and file_name as one string. Nested folders in the bucket, e.g. bucket1/folder/file1.sql are supported.
  • The bucket must be in the review project for your app. Appropriate IAM roles will be assumed by Workspace or CI Pipelines to download the file.
  • Two types of snapshots are supported, data or schema_and_data. This should be specified with snapshot_type at the service level of the yml, not the resource level.
    • Migrations will still be run in workspaces and non-production environments if data is specified. They will not be run if schema_and_data is specified. If defined, migrations always run in production pipelines, regardless of snapshot settings.
  • Snapshot loading in CI Pipelines will be injected until the environment has a pipeline with success status, after which it will not be loaded again.
  • You should export your data using the flags pg_dump --no-owner --no-privileges - the username used to access the database will likely differ on Coherence from your current deployment.

Use Existing Databases

If you have an existing database you'd like to use, you can point at it (rather than creating a new one) with yml like:

- name: db1
  engine: postgres
  version: 13
  type: database
  use_existing:
    project_type: review
    instance_name: EXISTING-INSTANCE-NAME
    manage_databases: true

• For GCP, supported project_types are production and review and correspond to the review_project_id and production_project_id configured in the Coherence dashboard for your application.
• In this configuration, manage_databases determines if Coherence will create and destroy databases on the instance to match the environments in your project. It generally should be true for review and false for production - since you won't need to madd/remove databases for preview environments in production.

For AWS this configuration is similar, but you'll need to create a secret containing your database username and password. RDS can manage the secret for you automatically during instance creation and manage it for you. The name of that secret should be added to your database configuration:

- name: db1
  engine: postgres
  version: 13
  type: database
  use_existing:
    project_type: production
    instance_name: EXISTING-DB-INSTANCE-IDENTIFIER
    db_password_secret: DB-PASSWORD-SECRET-NAME
    db_name: EXISTING-LOGICAL-DATABASE-NAME
    db_username: USERNAME-FROM-DB-PW-SECRET
    manage_databases: false

• If you are using Aurora, you should set the instance_name to the cluster name, and the additional configuration item cluster_mode to true.
• If you are using a multi-AZ cluster for Postgres/MySQL, you should also set use_db_proxy to false as they are not supported by AWS db proxy. You will need to add manually-managed variables to your environment to reach the read instance(s).

local_packages

Defines any paths in your repo that are installed to during container building, and which need to be copied to the host volume when mounting repo files into that container, for example in the cloud IDE.

• The most common use case here is an app that installs to node_modules.

Frontend Services

For frontend services, the container built from the Dockerfile at the root of repo_path will be used in Workspaces as the dev server for the service, and in CI Build Pipelines to build static assets to be served by the CDN.

build

Defines the command used to compile assets for serving. If not provided, defaults to yarn build.

build_path

This is the path in the repo where the build command will place compiled assets that we should copy and serve from the CDN. Common values are build, dist or public

dev

Defines the command used to run the web server in Workspaces

• The port 3000 is reserved by Coherence for the IDE service.
• The ports 80 and 8088 are reserved for internal Coherence use.

local_packages

Defines any paths in your repo that are installed to during container building, and which need to be copied to the host volume when mounting repo files into that container, for example in the cloud IDE.

• The most common use case here is an app that installs to node_modules.

CORS

You can define additional origins that can request your assets using CORS. See an example below:

cors:
    allowed_origins: ["www.example.com"]