Production Setup: High Availability/Multiconnect

This guide will first walk you through how to set up a High Availability cluster in production, then provide guidance on how to enable Multiconnect on top of it along with the changes required for a highly available Multiconnect cluster.

This document covers:

Prerequisites

It's recommended you set up a High Availability/Multiconnect WhatsApp Business API Client on a developer machine following the Developer Setup: High Availability and Multiconnect guide before following this guide to set up the WhatsApp Business API in production.

If you want to reuse the same phone number in production, please refer to the Migration guide before continuing with the rest of this guide. The content in this guide is based on the assumption of a fresh installation using a new phone number.

  1. Prepare multiple compute servers to run the various components of the Business API Client, including Webapp, Coreapp, and Master. Please refer to the What are the server system requirements to run the WhatsApp Business API Client? FAQ for the system requirements of the compute servers.

    For a production High Availability setup, we recommend at least 1 machine for the Webapp container, 2 machines for the Coreapp containers, and 2 machines for the Master containers. Loads on Masters are light and you could co-locate the Master and Coreapp containers on the same host.

  2. Install Docker on the compute servers.
  3. If Docker Compose is not bundled with your Docker installation, install it.
  4. Prepare a standalone database server running your MySQL process. The database server should not be run on any of the other compute servers hosting the Business API Client and should only be a few milliseconds of latency away from the other compute servers.

    MySQL 5.7xx or higher is required.

    Your database password should not contain any of these characters: ?{}&~!()^=

    Failing to comply with this will likely cause the setup to fail.

    There's also a known compatibility issue with MySQL 8. Please avoid using the latest for now, we're actively working on a fix.

  5. If you want to support sending/receiving media messages, a shared file system (e.g., NFS) is required with it mounted to a local directory on all Webapp, Master, and Coreapp hosts. Make sure read/write permissions are granted on the shared directory.
    mkdir /path-to-your-media-host
    mount -t nfs nfs_server_IP_addr:/shared_directory /path-to-your-media-host

Set up a Highly Available Cluster

  1. Create a biz directory to store the setup scripts:
    mkdir ~/biz; cd ~/biz;
  2. Clone the prod-multiconnect-compose.yml and db.env configuration files to ~/biz from the installation directory of the WhatsApp-Business-API-Setup-Scripts Github repo.
  3. Change the values in the db.env file to reflect your MySQL configuration:
    WA_DB_ENGINE=MYSQL
    WA_DB_HOSTNAME=your-database-server
    WA_DB_PORT=your-database-server-port
    WA_DB_USERNAME=your-database-username
    WA_DB_PASSWORD=your-database-password
  4. The whatsappMedia volume defined in the prod-multiconnect-compose.yml file is used for storing media files. By default, the volume in mounted to a directory on the Docker machine. Alternatively, if you have set up a shared file system for the Business API Client, change the volume line inside the service section of the prod-multiconnect-compose.yml file from whatsappMedia to the path of the host directory.
    services:
      waweb:
        ...
        volumes:
          - /path-to-your-media-host:/usr/local/wamedia
        ...
      wacore:
        ...
        volumes:
          - /path-to-your-media-host:/usr/local/wamedia
      ...
      master:
        ...
        volumes:
          - /path-to-your-media-host:/usr/local/wamedia
    
  5. To start up 1 Webapp container, 2 Master containers, and 2 Coreapp containers similar to the High Availability Introduction diagram, refer to the following commands and make necessary changes (i.e., hostnames, host usernames, and local paths) according to your environment:
    # copy configuration scripts to each Webapp host, ssh to each Webapp host, execute scripts to install Webapp on the host
    for host in your-webapp-hostname; do
        scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
        cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
        ssh username@$host $cmd waweb
    done 
    # copy configuration scripts to each master host, ssh to each master host, execute scripts to install master on the host
    for host in your-master1-hostname your-master2-hostname; do
        scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
        cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
        ssh username@$host $cmd master
    done
    # copy configuration scripts to each coreapp host, ssh to each coreapp host, execute scripts to install coreapp on the host
    for host in your-coreapp1-hostname your-coreapp2-hostname; do
        scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
        cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
        ssh username@$host $cmd wacore
    done
    
    In summary, the above commands will:
    • Copy the prod-multiconnect-compose.yml and db.env configuration files to all Webapp hosts (your-webapp-hostname in this example) and start the waweb service on these hosts;
    • Copy the prod-multiconnect-compose.yml and db.env configuration files to all Master hosts (your-master1-hostname and your-master2-hostname in this example) and start the Master service on these hosts;
    • Copy the prod-multiconnect-compose.yml and db.env configuration files to all Coreapp hosts (your-coreapp1-hostname and your-coreapp2-hostname in this example) and start the wacore service on these hosts.
  6. Check containers are in RUNNING status on all hosts by running the following commands.
    EXTERNAL_HOSTNAME=$host docker-compose -f prod-multiconnect-compose.yml ps
    
  7. You can download and configure our Postman Collection to easily interact with the WhatsApp Business API.
  8. Perform a health check to verify the WhatsApp Business API is running properly. Example response:
    {
        "health": {
            "your-master1-hostname:85cdd51506fd": {
                "errors": [
                  {
                      "code": 1011,
                      "title": "Service not ready",
                      "details": "Wacore is not instantiated. Please check wacore log for details."
                  }
                ]
            },
            "your-master2-hostname:8dd3f5bea27d": {
                "gateway_status": "unregistered",
                "role": "primary_master"
            },
            "your-coreapp1-hostname:753efb1cf72c": {
                "errors": [
                  {
                      "code": 1011,
                      "title": "Service not ready",
                      "details": "Wacore is not instantiated. Please check wacore log for details."
                  }
                ]
            },
            "your-coreapp2-hostname:75d7355eaaaa": {
                "errors": [
                  {
                      "code": 1011,
                      "title": "Service not ready",
                      "details": "Wacore is not instantiated. Please check wacore log for details."
                  }
                ]
            }
        }
    }
    200
    
    Because your Business API Client is not registered right now, you will see primary_master has an unregistered gateway_status in the health status response.
  9. Follow the Registration documentation to register the WhatsApp Business API client.
  10. Perform another health check after registration to make sure that one of the Coreapps is now in connected status. Example response:
    {
        "health": {
            "your-master1-hostname:85cdd51506fd": {
                "gateway_status": "disconnected",
                "role": "secondary_master"
            },
            "your-master2-hostname:8dd3f5bea27d": {
                "gateway_status": "disconnected",
                "role": "primary_master"
            },
            "your-coreapp1-hostname:753efb1cf72c": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "your-coreapp2-hostname:75d7355eaaaa": {
                "gateway_status": "disconnected",
                "role": "coreapp"
            }
        }
    }
    200
    
    Note: In High Availability mode, only one Coreapp (your-coreapp1-hostname:753efb1cf72c in this example) will be connected to the Whatsapp server, all other nodes including the primary Master will have a gateway_status of disconnected. If your-coreapp1-hostname:753efb1cf72c goes down, your-coreapp2-hostname:75d7355eaaaa will replace it and connect to the Whatsapp server to maintain High Availability.
  11. It's recommended to set up monitoring for your production Business API Client, see the Monitoring and Alerting documentation for more information.
  12. Now, you have set up the WhatsApp Business API in High Availability mode. In this mode, only one Coreapp is able to connect to the WhatsApp server to send messages at any given time. If you want to have multiple Coreapps sending messages at the same time to increase message throughput, follow the steps in the Set up a Highly Availability Multiconnect Cluster section below.

Set up a Highly Available Multiconnect cluster

  1. Use the shards API endpoint to set up 2 shards. You should receive a 201 Created HTTP status code upon successful setting of shards.
  2. Perform a health check to verify all nodes are running properly. Example response:
    {
        "health": {
            "your-master1-hostname:85cdd51506fd": {
                "gateway_status": "disconnected",
                "role": "secondary_master"
            },
            "your-master2-hostname:8dd3f5bea27d": {
                "gateway_status": "connected",
                "role": "primary_master"
            },
            "your-coreapp1-hostname:753efb1cf72c": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "your-coreapp2-hostname:75d7355eaaaa": {
                "gateway_status": "connected",
                "role": "coreapp"
            }
        }
    }      
    200    
    
    Note: In Multiconnect mode with 2 shards, 2 Coreapps (your-coreapp1-hostname:753efb1cf72c and your-coreapp2-hostname:75d7355eaaaa in this example) will be connected to the WhatsApp server. Different from High Availability, the primary Master (your-master2-hostname:8dd3f5bea27d in this example) will also connect to the server.
  3. So far in this example, you have 2 Coreapp containers and message loads are split between them. However, if one of the Coreapp containers goes down, half of message sends will fail. In order to maintain High Availability in this new Multiconnect setup, you can start a third Coreapp on a new Coreapp host (your-coreapp3-hostname in this example) to tolerate 1 Coreapp failure, which is similar to the diagram shown in the Multiconnect Introduction. To start the third Coreapp container, run the following command:
    # copy configuration scripts to the 3rd Coreapp host, ssh to the Coreapp host, execute scripts to install Coreapp on the host
    for host in your-coreapp3-hostname; do
        scp db.env prod-multiconnect-compose.yml username@$host:/local/path/
        cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
        ssh username@$host $cmd wacore
    done
    
  4. Perform a health check to verify all nodes are running properly. Example response:
    {
        "health": {
            "your-master1-hostname:85cdd51506fd": {
                "gateway_status": "disconnected",
                "role": "secondary_master"
            },
            "your-master2-hostname:8dd3f5bea27d": {
                "gateway_status": "disconnected",
                "role": "primary_master"
            },
            "your-coreapp1-hostname:753efb1cf72c": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "your-coreapp2-hostname:75d7355eaaaa": {
                "gateway_status": "connected",
                "role": "coreapp"
            },
            "your-coreapp3-hostname:23b50199bec2": {
                "gateway_status": "disconnected",
                "role": "coreapp"
            }
        }
    }      
    200 
    
    The new Coreapp container (your-coreapp3-hostname:23b50199bec2 in this example) now acts as a standby container and is not currently connected to the WhatsApp server. If either of the other 2 connected Coreapp containers stops working, the third container will connect to the WhatsApp server to maintain an overall shard count of 2.

Guidelines & Best Practices

Number of Webapps

To support 100-150 messages/sec, at least 2 Webapps are recommended. If you want to achieve High Availability for Webapp, you could run Webapp containers on more than 2 hosts and host them behind a load balancer such as HAProxy, Nginx, or ELB. Instead of accessing the Business API Client endpoints through https://your-webapp-hostname:your-webapp-port/, you should use https://your-load-balancer-name:your-load-balancer-port/.

Number of Masters

It's recommended to have 3 Masters running on different hosts. There is no reason to have more than 3 Masters in production no matter how many shards you have. Loads on Masters are light and you could co-locate them with Coreapp containers.

Number of Coreapps

It's recommended to have shard_number + X number of Coreapps running on different hosts in order to tolerate X number of host failures.

Upgrading

To upgrade an installation, use WA_API_VERSION to upgrade. For example:

# ssh to each Webapp host, execute scripts with new WA_API_VERSION to upgrade Webapp on the host
for host in your-webapp-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.21.6 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd waweb
done
# ssh to each Master host, execute scripts with new WA_API_VERSION to upgrade Master on the host
for host in your-master1-hostname your-master2-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.21.6 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd master
done
# ssh to each Coreapp host, execute scripts with new WA_API_VERSION to upgrade Coreapp on the host
for host in your-coreapp1-hostname your-coreapp2-hostname your-coreapp3-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host WA_API_VERSION=2.21.6 docker-compose -f /local/path/prod-multiconnect-compose.yml up -d"
    ssh username@$host $cmd wacore
done

Uninstalling

If you need to reset your development environment by removing all containers, run the following command:

# ssh to each Webapp host, execute scripts to uninstall Webapp on the host
for host in your-webapp-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-docker-compose.yml down"
    ssh username@$host $cmd waweb
done
# ssh to each Master host, execute scripts to uninstall Master on the host
for host in your-master1-hostname your-master2-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-docker-compose.yml down"
    ssh username@$host $cmd master
done
# ssh to each Coreapp host, execute scripts to uninstall Coreapp on the host
for host in your-coreapp1-hostname your-coreapp2-hostname your-coreapp3-hostname; do
    cmd="EXTERNAL_HOSTNAME=$host docker-compose -f /local/path/prod-docker-compose.yml down"
    ssh username@$host $cmd wacore
done

Troubleshooting

To collect logs from all containers on a host, run the following command:

EXTERNAL_HOSTNAME=$host_to_collect_logs docker-compose -f /local/path/prod-multiconnect-compose.yml logs > debug_output.txt

To collect logs of a specific service, append the service name (e.g., waweb, master1, wacore1) to the docker-compose logs command. For example:

EXTERNAL_HOSTNAME=$host_to_collect_logs docker-compose -f /local/path/prod-multiconnect-compose.yml logs waweb > debug_output.txt

You can find the logs in the debug_output.txt file in the current directory.