Data Management

This document discusses how to manage the various different types of data and databases associated with the WhatsApp Business API client.

Volumes

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. The WhatsApp Business API client requires one Docker volume for media. This volume is created automatically when running the docker-compose commands during installation.

Docker volumes support storing volumes on remote hosts or cloud providers. You can set up the media volume on any existing datacenter fileshare solution. Make sure all the Webapp and Coreapp containers have read/write access to it. To view or edit the path to the volume, edit the docker-compose.yml file.

NameDescription

Media Volume

  • Stores incoming and outgoing media files
  • Mounts within containers, by default, at /usr/local/wamedia/

Environment Variables

The WhatsApp Business API client can use MySQL or PostgreSQL to store data.

You can configure the database settings by setting the following environment variables in the db.env file when installing the WhatsApp Business API client. These environment variables are used by the Coreapp and Webapp when connecting to the database.

Database Setting Environment Variable

database_engine

WA_DB_ENGINE

hostname

WA_DB_HOSTNAME

port

WA_DB_PORT

username

WA_DB_USERNAME

password

WA_DB_PASSWORD

database_name_prefix

WA_DB_NAME_PREFIX

connection_idle_timeout

WA_DB_CONNECTION_IDLE_TIMEOUT

  • WA_DB_NAME_PREFIX — Can be used to prefix all databases created when the WhatsApp Business API client is installed. It can be used to run multiple sets of WhatsApp Business API databases on the same database host.
  • WA_DB_CONNECTION_IDLE_TIMEOUT (only supported by MySQL) — By setting this environment variable (in milliseconds) while starting the Coreapp, you can set the idle timeout for your MySQL database. The MySQL server will then close any database connections that are idle for the time set.

Database Management

These are recommendations for managing the database associated with your WhatsApp Business API client.

ConcernRecommendations

Upgrading with a large database

Large databases are considered those with more than 2 million rows in the messages table.
Use the recommendations below for using the /services/message/gc endpoint during an upgrade to maintain database stability.

Beginning with v2.29.1, the pass_through parameter is set to false by default, which may cause an increase in database storage for the messages table.

Use the automatic garbage collection parameter to ensure your database operates with stability by cleaning up the database periodically. See the Application Settings documentation for more information on the pass_through and db_garbagecollector_enable parameters.

Automatic garbage collection configuration

  • Enabled for businesses who upgrade with the pass_through parameter enabled.
  • Disabled for businesses who upgrade with the pass_through parameter disabled.
    • If pass_through was disabled before upgrading, it is recommended you enable automatic garbage collection after upgrade using the db_garbagecollector_enable parameter in the Application Settings
  • For businesses with traffic > 120 qps, API call-based garbage collection with the /services/message/gc endpoint is recommended.

API call-based garbage collection

  • Makes database cleanup integration-dependent and is, ideally, to be avoided for most businesses.
  • It is recommended that businesses using the WhatsApp Business API to perform database garbage collection:
    • Mark incoming messages as read before making the API request to enable aggressive cleanup.
    • We recommend the `/services/message/gc endpoint be used once every 24 hours during a downtime (i.e., low outgoing message volume).
    • For a messages table with more than 10 million rows, you may need to run the API request multiple times to ensure the callback notifications return with no errors.

See the Services documentation for more information.

Log Rotation

A log rotation script is packaged within the Coreapp and Webapp containers.

Webapp

The log rotation script in the Webapp container:

  • Uses the logrotate utility
  • Keeps the last 30 files of web, access, and error logs
  • Rotates only if the size is greater than 20MB
  • Compresses rotated log files
  • Archives old log files to /var/log/whatsapp/archive

Coreapp

The log rotation script in the Coreapp container:

  • Uses an in-house script
  • Keeps the last 30 files
  • Rotates automatically if size is greater than 15MB
  • Compresses rotated log files
  • Archives old log files to /var/log/whatsapp/archive

In the Coreapp, a new log file is only created when the size exceeds 15MB per log file. The old log files are not removed automatically.

Cleanup Script Recommendation

It is recommended you periodically (i.e., daily) execute the cleanup script below to perform log rotation on all Webapp, Coreapp and Master (in the case of a Multiconnect setup) containers. It's best to configure a cronjob on your host to perform on all running WhatsApp Business API containers and be executed during off-peak hours. Invoking the cleanup script periodically keeps the disk space consumed by log files under control.

docker exec your-container-name /opt/whatsapp/bin/cleanup.sh

FAQ

MySQL 5.7.x, PostgreSQL 9.5.x, 9.6.x, 10.x are required. Using a previous version will throw an Unable to initialize config store error.

Set up MySQL locally using Docker by following the Docker MySQL guide.

Set up PostgreSQL locally using Docker by following the Docker PostgreSQL guide.

In most cases you should run the database on a separate physical server from the core and web containers. The database server should only be a few milliseconds of latency away from the compute machine(s).

MySQL and PostgreSQL are supported. If you run Docker by yourself, you must provide a MySQL/PostgreSQL database for the containers to connect to. Using the AWS template sets up a MySQL database by default.

No. Currently, the WhatsApp Business API client does not run on Docker for Windows. For development needs, using a Linux Virtual machine and running Docker within it is the recommended solution. For production workloads, we recommend using a Linux Server to avoid compatibility and performance issues.

You can restart the Docker containers by running the following code:

Coreapp Docker container

docker restart wacore<Current_WABA_Version>

Webapp Docker container

docker restart webapp<Current_WABA_Version>

You can check which version you're running with

docker ps

Yes, log rotation for webapp containers and coreapp containers has slightly different behaviors:

  • Webapp: the last 30 log files will be retained. The log file is rotated only if its size is greater than 20MB.
  • Coreapp: the last 30 log files will be retained. The log file is rotated only if its size is greater than 15MB. Rotated files are compressed.

There is a script that can be triggered externally to clean up old logs of a container:

docker exec CONTAINER_NAME /opt/whatsapp/bin/cleanup.sh

The script works with both webapp and coreapp containers. By running the script, old log files will be removed so that only 30 log files of the container remain.

Your system might start slowing down as the space fills up. This can be caused by many media files, messages, and large log files. Log files are automatically rotated, but if they start to get large, deleting them is safe.

Messages are stored in the database. You can delete messages as necessary. Also, if the pass_through is set to false in the application settings, then all messages are saved in the database until they are explicitly deleted.

Media files that users send to you are downloaded to the media volumes. It is up to the business to decide which media files to delete, but it is generally safe to delete any media files. You can use docker inspect your-container-id to check where the media volume folder is.

Yes, the database can be used in other ways without touching the WhatsApp related tables.

The database tables store information related to app settings, chat threads, messages, media, etc., which are all required by the app to operate.

v2.25.x improves outbound and inbound performance over previous releases. This optimization relies on creating additional database connections. For some deployments, this can cause the number of database connections to increase and reach configured limits. To keep the increased performance, you can increase the maximum number of connections your database server can accept. If that's not possible, you can change the the axolotl_context_striping_disabled parameter to disable this behavior. See the Applications Settings documentation for more information on how to do this change.

Database garbage collection periodically cleans up the messages and messages_reciept_log tables to help with database management.

The garbage collector retains certain messages to allow for successful delivery/processing. For example, retaining incoming message for a certain period of time to allow business integrations to mark the message as read.

The Coreapp performs garbage collection at random intervals (i.e., every few hours). This is to prevent potential performance degradation in High Availability stacks due to database contention.

Garbage collection is independent of the callback queue. For example, if the Webhook server isn't available for 4 days, the callbacks will be stored to be delivered when the Webhook server connectivity is restored.

Use the database garbage collection services API endpoint to purge messages and corresponding message receipts from the messageStore.messages and messageStore.messages_receipt_log tables.