Developer Setup: High Availability and Multiconnect

This guide will walk you through how to first set up a High Availability cluster on a developer machine, then provide guidance on how to enable Multiconnect on top of it along with the changes required for a highly available Multiconnect cluster.

This document covers:

Prerequisites

If you have followed the Developer Setup: Single Instance guide to set up WhatsApp Business API Client on a developer machine previously, please follow the Migration guide before continuing with the rest of this guide. The content in this guide is based on the assumption of a fresh installation.

For production setup, follow the instructions in Production Setup guides.

  1. Install Docker on your system.
  2. If Docker Compose is not bundled with your Docker installation, install it.
  3. It's recommended you set up a test account in a development environment to test new releases.
  4. It's highly recommended to read through the Availability and Scaling guide to learn more about High Availability/Multiconnect.

Set up a High Availability Cluster

  1. Create a biz directory to store the setup scripts:
    mkdir ~/biz; cd ~/biz;
  2. Get the configuration files from the WhatsApp-Business-API-Setup-Scripts Github repo.
  3. Set the WA_API_VERSION environment variable to the current version using:
    export WA_API_VERSION=new-whatsapp-version
  4. To start up a High Availability cluster with 1 database container, 1 Webapp container, 2 Master containers and 2 Coreapp containers in the background similar to the diagram shown in the High Availability Introduction, run the following command:
    docker-compose -f multiconnect-compose.yml up -d db waweb master1 master2 wacore1 wacore2
    Example output:
    Creating network "biz_default" with the default driver
    Creating volume "biz_mysqlData" with local driver
    Creating volume "biz_whatsappMedia" with local driver
    Creating biz_db_1 ... done
    Creating biz_waweb_1 ... done
    Creating biz_master1_1 ... done
    Creating biz_master2_1 ... done
    Creating biz_wacore2_1 ... done
    Creating biz_wacore1_1 ... done
    
  5. Check that all containers have an UP state by running:
    docker-compose -f multiconnect-compose.yml ps
    Example output:
          Name                    Command               State                                                 Ports
    --------------------------------------------------------------------------------------------------------------------------------------------------------------
    biz_db_1        docker-entrypoint.sh mysqld      Up      0.0.0.0:33060->3306/tcp, 33060/tcp
    biz_master1_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32931->6250/tcp, 0.0.0.0:32930->6251/tcp, 0.0.0.0:32928->6252/tcp, 0.0.0.0:32926->6253/tcp
    biz_master2_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32929->6250/tcp, 0.0.0.0:32927->6251/tcp, 0.0.0.0:32925->6252/tcp, 0.0.0.0:32924->6253/tcp
    biz_wacore1_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32937->6250/tcp, 0.0.0.0:32935->6251/tcp, 0.0.0.0:32933->6252/tcp, 0.0.0.0:32932->6253/tcp
    biz_wacore2_1   /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:32939->6250/tcp, 0.0.0.0:32938->6251/tcp, 0.0.0.0:32936->6252/tcp, 0.0.0.0:32934->6253/tcp
    biz_waweb_1     /opt/whatsapp/bin/wait_on_ ...   Up      0.0.0.0:9090->443/tcp
    
    By default, the Webapp container will be running on port 9090 and the database container will be running on port 33060.
  6. You can download and configure our Postman Collection to easily interact with the WhatsApp Business API.
  7. Perform a health check to verify all nodes are running properly. Example response:
    {
        "health": {
          "master1:b28d835cd579": {
              "errors": [
                  {
                      "code": 1011,
                      "title": "Service not ready",
                      "details": "Wacore is not instantiated. Please check wacore log for details."
                  }
              ]
          },
          "master2:7fe542d305b4": {
              "gateway_status": "unregistered",
              "role": "primary_master"
          },
          "wacore1:35a5fabfc79d": {
              "errors": [
                  {
                      "code": 1011,
                      "title": "Service not ready",
                      "details": "Wacore is not instantiated. Please check wacore log for details."
                  }
              ]
          },
          "wacore2:05e1a6d70665": {
              "errors": [
                  {
                      "code": 1011,
                      "title": "Service not ready",
                      "details": "Wacore is not instantiated. Please check wacore log for details."
                  }
              ]
          }
      }
    }
    
    Because your WhatsApp Business API Client is not registered right now, you will see primary_master has an unregistered gateway_status in the health status response.
  8. Follow the Registration documentation to register the WhatsApp Business API cilent.
  9. Perform another health check after registration to make sure that one of the coreapps is now in connected status. Example response:
    {
        "health": {
            "master1:b28d835cd579": {
                "gateway_status": "disconnected",
                "role": "secondary_master"
            },
            "master2:7fe542d305b4": {
                "gateway_status": "disconnected",
                "role": "primary_master"
            },
            "wacore1:35a5fabfc79d": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "wacore2:05e1a6d70665": {
                "gateway_status": "disconnected",
                "role": "coreapp"
            }
        }
    } 
    
    Note: In High Availability mode, only one Coreapp (wacore1 in this example) will be connected to the Whatsapp server, all other nodes including the primary master will have a gateway_status of disconnected. If wacore1 goes down, wacore2 will replace it and connect to the Whatsapp server to maintain High Availability.

Now, you have set up the WhatsApp Business API in High Availability mode. In this mode, only one Coreapp is able to connect to the WhatsApp server to send messages at any given time. If you want to have multiple Coreapps sending messages at the same time to increase message throughput, follow the steps in the Set up a Highly Availability Multiconnect Cluster section below.

Set up a Highly Available Multiconnect Cluster

  1. Use the shards API endpoint to set up 2 shards, and you should see an HTTP response with status 201 Created.
  2. Perform a health check to verify all nodes are running properly. Example response:
    {
        "health": {
            "master1:b28d835cd579": {
                "gateway_status": "disconnected",
                "role": "secondary_master"
            },
            "master2:7fe542d305b4": {
                "gateway_status": "connected",
                "role": "primary_master"
            },
            "wacore1:35a5fabfc79d": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "wacore2:05e1a6d70665": {
                "gateway_status": "connected",
                "role": "coreapp"
            }
        }
    }          
    
    Note: In Multiconnect mode with 2 shards, 2 Coreapps (wacore1 and wacore2 in this example) will be connected to the WhatsApp server, and the primary Master (master2 in this example) will also connect to the WhastsApp server.

  3. So far in this example, you have 2 Coreapp containers and message loads are split between them. However, if one of the Coreapp containers goes down, half of message sends will fail. In order to maintain High Availability in this new Multiconnect setup, you can start a third Coreapp to tolerate 1 Coreapp failure, which is similar to the diagram shown in the Multiconnect Introduction. To start the third Coreapp container, run the following command:
    docker-compose -f multiconnect-compose.yml up -d wacore3
    Example output:
    biz_db_1 is up-to-date
    biz_waweb_1 is up-to-date
    biz_master1_1 is up-to-date
    Creating biz_wacore3_1 ... done
    
  4. Perform a health check to verify all nodes are running properly. The new wacore3 container now acts as a standby container and is not currently connected to the WhatsApp server. If either wacore1 or wacore2 stops working, wacore3 will connect to the WhatsApp server to maintain an overall shard count of 2. Example response:
    {
        "health": {
            "master1:b28d835cd579": {
                "gateway_status": "disconnected",
                "role": "secondary_master"
            },
            "master2:7fe542d305b4": {
                "gateway_status": "connected",
                "role": "primary_master"
            },
            "wacore1:35a5fabfc79d": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "wacore2:05e1a6d70665": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "wacore3:23b50199bec2": {
                "gateway_status": "disconnected",
                "role": "coreapp"
            }
        }
    } 
    

Upgrading

It is highly recommended to backup your current application settings before upgrading. Please follow the Backup and Restore documentation.

To upgrade an installation, change the WA_API_VERSION environment variable to the new version and restart the Docker containers:

export WA_API_VERSION=new-whatsapp-version
  
docker-compose -f multiconnect-compose.yml up -d

For MySQL database users upgrading to version 2.23.x and above

You can now make use of a database upgrade service that will let you upgrade your database while your application is still running to avoid downtime*.

  1. Download the dbupgrade-compose.yml file, which has fields indicating the container version.
    For example:
    services:
      dbupgrade:
        image: docker.whatsapp.biz/coreapp:v${WA_API_VERSION:-2.21.3}
    
  2. To upgrade an installation, run the dbupgrade-service container with the WA_API_VERSION environment variable set to the latest version:
    WA_API_VERSION=2.23.4 docker-compose -f dbupgrade-compose.yml up -d
    
    Note: If you are using an orchestration that restarts the container upon exit irrespective of the exit code, start the service with the EXIT_ON_SUCCESS environment variable set to FALSE in order to avoid exiting the container when the exit code is 0.
  3. Wait for the upgrade to finish. If the database upgrade is successful, the container will exit with code 0. You can use the following Docker command to track the status:
    docker wait your-database-upgrade-container-name
    This will output the exit code of the dbupgrade-service container.
  4. Restart the Coreapp and Webapp Docker containers with the WA_API_VERSION environment variable set to the latest version:
    WA_API_VERSION=2.23.4 docker-compose -f multiconnect-compose.yml up -d
    

* If you're upgrading from 2.19.7, you may still experience some downtime. It is always recommended to perform upgrades during your least busiest hours.

Uninstalling

If you need to reset your development environment by removing all containers, run the following command from the directory containing the multiconnect-compose.yml file:

docker-compose -f multiconnect-compose.yml down

In order to get rid of all volumes defined in the multiconnect-compose.yml file in addition to the containers, run down with the -v parameter:

docker-compose -f multiconnect-compose.yml down -v

Troubleshooting

To collect logs from all containers, run the following command:

docker-compose -f multiconnect-compose.yml logs > debug_output.txt

To collect logs of a specific service, append the service name (e.g.: waweb, master1, wacore1) to the docker-compose logs command, e.g.:

docker-compose -f multiconnect-compose.yml logs waweb > debug_output.txt

You can find the logs in the debug_output.txt file in the current directory.