Data Management

This document discusses how to manage the various different types of data and databases associated with the WhatsApp Business API client.

Volumes

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. The WhatsApp Business API client requires one Docker volume for media. This volume will be created automatically when running the docker-compose commands during installation.

Docker volumes support storing volumes on remote hosts or cloud providers. You can set up the media volume on any existing datacenter fileshare solution. Make sure all the Webapp and Coreapp containers have read/write access to it. To view or edit the path to the volume, edit the docker-compose.yml file.

NameDescription

Media Volume

  • Stores incoming and outgoing media files
  • Mounts within containers, by default, at /usr/local/wamedia/

MySQL

The WhatsApp Business API client can use MySQL to store data.

Environment Variables

You can configure the database settings by setting the following environment variables in the db.env file when installing the WhatsApp Business API client. These environment variables will be used by the Coreapp and Webapp when connecting to the MySQL database.

Database Setting Environment Variable

database_engine

WA_DB_ENGINE

hostname

WA_DB_HOSTNAME

port

WA_DB_PORT

username

WA_DB_USERNAME

password

WA_DB_PASSWORD

database_name_prefix

WA_DB_NAME_PREFIX

connection_idle_timeout

WA_DB_CONNECTION_IDLE_TIMEOUT

  • WA_DB_NAME_PREFIX — Can be used to prefix all databases created when the WhatsApp Business API client is installed. It can be used to run multiple sets of WhatsApp Business API databases on the same database host.
  • WA_DB_CONNECTION_IDLE_TIMEOUT — By setting this environment variable (in milliseconds) while starting the Coreapp, you can set the idle timeout for your MySQL database. The MySQL server will then close any database connections that are idle for the time set.

Database Management

These are recommendations for managing the database associated with your WhatsApp Business API client.

ConcernRecommendations

Upgrading with a large database

Large databases are considered those with more than 2 million rows in the messages table.
Use the receommendations below for using the /services/message/gc endpoint during an upgrade to maintain database stability.

Beginning with v2.29.1, the pass_through parameter is set to false by default, which may cause an increase in database storage for the messages table.

Use the automatic garbage collection parameter to ensure your database operates with stability by cleaning up the database periodically. See the Application Settings documentation for more information on the pass_through and db_garbagecollector_enable parameters.

Automatic garbage collection configuration

  • Enabled for businesses who upgrade with the pass_through parameter enabled.
  • Disabled for businesses who upgrade with the pass_through parameter disabled.
    • If pass_through was disabled before upgrading, it is recommended you enable automatic garbage collection after upgrade using the db_garbagecollector_enable parameter in the Application Settings
  • For businesses with traffic > 120 qps, API call-based garbage collection with the /services/message/gc endpoint is recommended.

API call-based garbage collection

  • Makes database cleanup integration-dependent and is, ideally, to be avoided for most businesses.
  • It is recommended that businesses using the WhatsApp Business API to perform database garbage collection:
    • Mark incoming messages as read before making the API request to enable aggressive cleanup.
    • We recommend the `/services/message/gc endpoint be used once every 24 hours during a downtime (i.e., low outgoing message volume).
    • For a messages table with more than 10 million rows, you may need to run the API request multiple times to ensure the callback notifications return with no errors.

See the Services documentation for more information.

Log Rotation

A log rotation script is packaged within the Coreapp and Webapp containers.

Webapp

The log rotation script in the Webapp container:

  • Uses the logrotate utility,
  • Keeps the last 30 files of web, access, and error logs,
  • Rotates only if the size is greater than 20MB,
  • Compresses rotated log files, and
  • Archives old log files to /var/log/whatsapp/archive.

Coreapp

The log rotation script in the Coreapp container:

  • Uses an in-house script,
  • Keeps the last 30 files,
  • Rotates automatically if size is greater than 15MB,
  • Compresses rotated log files, and
  • Archives old log files to /var/log/whatsapp/archive.

In the Coreapp, a new log file is only created when the size exceeds 15MB per log file. The old log files are not removed automatically.

Cleanup Script Recommendation

It is recommended you periodically (i.e., daily) execute the cleanup script below to perform log rotation on all Webapp, Coreapp and Master (in the case of a Multiconnect setup) containers. It's best to configure a cronjob on your host to perform on all running WhatsApp Business API containers and be executed during off-peak hours. Invoking the cleanup script periodically will keep the disk space consumed by log files under control.

docker exec your-container-name /opt/whatsapp/bin/cleanup.sh