Usage with Gitlab CI CD

We use Gitlab CI/CD to run Masquerade nightly and create an off-site anonymized backup. We have a project group for each client, and inside that group we create a new project called 'anonymize' (you may call this whatever you like).

We fetch the uploaded database from a S3 bucket (it is uploaded nightly from the production server), import it into a database, anonymize it and re-upload it to the S3 bucket. This way, the anonymization procedure happens totally separate from the production server, eliminating the off-chance we accidentally anonymize the production database.

In this project, we have one file; .gitlab-ci.yml. This is its contents:

image: path-to-our-masquerade-container

database-anonymization:
    tags:
        - database
    
    services:
        - mysql:5.6.37
        
    before_script:
        - mysql --version
        - php -v
        
    script:
        - date

        # Configure aws
        - mkdir -p ~/.aws
        - printf "[default]\naws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}\naws_access_key_id = ${AWS_ACCESS_KEY_ID}" > ~/.aws/credentials
        - printf "[default]\nregion = ${AWS_DEFAULT_REGION}" > ~/.aws/config
        
        # Configure MySQL
        - printf "[client]\nuser = ${MYSQL_USER}\npassword = ${MYSQL_ROOT_PASSWORD}\nhost = mysql" > ~/.my.cnf; date
        
        # Anonymize database
        - ~/.local/bin/aws s3 cp s3://s3-bucket-here/stripped.sql - | grep -v INFORMATION_SCHEMA.SESSION_VARIABLES | sed -s 's/ROW_FORMAT=\"*FIXED\"*//' | mysql ${MYSQL_DATABASE}; date
        - masquerade run --prefix=${PREFIX} --platform=magento2  --host=mysql --database=${MYSQL_DATABASE} --username=${MYSQL_USER} --password=${MYSQL_ROOT_PASSWORD} --locale=${LOCALE}; date
        - mysqldump ${MYSQL_DATABASE} > anon.sql; date
        - ~/.local/bin/aws s3 mv anon.sql s3://s3-bucket-here/anon.sql; date
        
        # Clean up
        - echo "DROP DATABASE IF EXISTS ${MYSQL_DATABASE}" | mysql; date
        - rm -rf ~/.aws
        - rm -rf ~/.my.cnf

We use a Docker container for Masquerade - it also contains MySQL for the database to be anonymized in. This is the Dockerfile for it:

FROM romeoz/docker-apache-php:7.2
MAINTAINER Peter Jaap Blaakmeer <[email protected]>

RUN apt-get update

# Install awscli
RUN apt-get install -y libpython-dev python-dev libyaml-dev python-pip
RUN pip install awscli --upgrade --user

# Install mysql-client
RUN apt-get install -y mysql-client

# Install masquerade
RUN curl -LO https://github.com/elgentos/masquerade/releases/latest/download/masquerade.phar
RUN chmod +x ./masquerade.phar && mv ./masquerade.phar /usr/bin/masquerade

# Run original image's entrypoint manually
CMD ["/sbin/entrypoint.sh"]

Now set the following variables in Gitlab's project Settings > CI/CD > Variables with your values;

• MYSQL_USER
• MYSQL_ROOT_PASSWORD
• MYSQL_DATABASE
• PREFIX
• LOCALE

And configure a nightly pipeline run under CI / CD > Schedules.