Skip to content

Commit

Permalink
Update test doc
Browse files Browse the repository at this point in the history
  • Loading branch information
@enyachoke
enyachoke committed Nov 5, 2024
1 parent 7682500 commit 67b8380
Show file tree
Hide file tree
Showing 4 changed files with 37 additions and 37 deletions.
70 changes: 35 additions & 35 deletions Testing.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
## Testing Backup and Restore
To test the project we need to simulate running a live system , taking it down and restoring the system. For this test we have two Docker Compose files for the system `docker-compose-live.yaml` and `docker-compose-restore-test.yaml` in a real restore scenario we will only have on Docker Compose file as we expect the restore is being done on a separate instance. We also have separate files for Backup and Restore (`docker-compose-backup-service.yaml` and `docker-compose-restore-service.yaml`)
To test the project we need to simulate running a live system , taking it down and restoring the system. For this test we have two Docker Compose files for the system `docker-compose-live.yaml` and `docker-compose-live-two.yaml` in a real restore scenario we will only have on Docker Compose file as we expect the restore is being done on a separate instance. We also have separate files for Backup and Restore (`docker-compose-backup-service.yaml` and `docker-compose-restore-service.yaml`)
Run:

```./start-live-system.sh```
```./start-live.sh```
This will start the example live system;
Wait for > 5 minutes for the first backup to happen. You can confirm by running

`docker logs -f live-system-backup-1`
`docker logs -f live-backup-1`

You should see logs similar to:

```
crond 4.5 dillon's cron daemon, started with loglevel notice
2024-10-21 03:50:01,338 - INFO: Starting backup container
2024-10-21 03:50:01,647 - INFO: Backup process container: compassionate_ganguly
2024-10-21 03:50:01,848 - INFO: 2024-10-21 03:50:01,848 - INFO: Status for compose project 'live-system'
2024-10-21 03:50:01,848 - INFO: 2024-10-21 03:50:01,848 - INFO: Status for compose project 'live'
2024-10-21 03:50:01,848 - INFO: 2024-10-21 03:50:01,848 - INFO: Repository: '/restic_data'
2024-10-21 03:50:01,849 - INFO: 2024-10-21 03:50:01,848 - INFO: Backup currently running?: True
2024-10-21 03:50:01,849 - INFO: 2024-10-21 03:50:01,848 - INFO: Include project name in backup path?: False
Expand All @@ -28,28 +28,28 @@ crond 4.5 dillon's cron daemon, started with loglevel notice
2024-10-21 03:50:03,933 - INFO: 2024-10-21 03:50:03,933 - INFO: Successfully initialized repository: /restic_data
2024-10-21 03:50:03,933 - INFO: 2024-10-21 03:50:03,933 - INFO: ------------------------- Detected Config -------------------------
2024-10-21 03:50:03,934 - INFO: 2024-10-21 03:50:03,933 - INFO: service: postgres
2024-10-21 03:50:03,934 - INFO: 2024-10-21 03:50:03,934 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,939 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,934 - INFO: 2024-10-21 03:50:03,934 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,939 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,940 - INFO: - postgres (is_ready=True) -> /backup/databases/postgres/test-postgres.sql
2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,940 - INFO: service: mariadb
2024-10-21 03:50:03,941 - INFO: 2024-10-21 03:50:03,940 - INFO: Container NAME live-system-mariadb-1
2024-10-21 03:50:03,941 - INFO: 2024-10-21 03:50:03,940 - INFO: Container NAME live-mariadb-1
2024-10-21 03:50:03,948 - INFO: 2024-10-21 03:50:03,948 - INFO: - mariadb (is_ready=True) -> /backup/databases/mariadb/all_databases.sql
2024-10-21 03:50:03,948 - INFO: 2024-10-21 03:50:03,948 - INFO: service: web
2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: - volume: /workspace/restic-compose-backup/src/tests -> /backup/volumes/web/srv/tests
2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: -------------------------------------------------------------------
2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: Backing up databases
2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: Backing up postgres in service postgres
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Backing PostgresContainer container postgres /backup/databases/postgres/test-postgres.sql
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-system-postgres-1
2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-postgres-1
2024-10-21 03:50:04,145 - INFO: 2024-10-21 03:50:04,144 - INFO: Database backup exit code: =======> 0
2024-10-21 03:50:04,145 - INFO: 2024-10-21 03:50:04,145 - INFO: Exit code: 0
2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Backing up mariadb in service mariadb
2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-system-mariadb-1
2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-system-mariadb-1
2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-mariadb-1
2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-mariadb-1
2024-10-21 03:50:04,348 - INFO: 2024-10-21 03:50:04,347 - INFO: Database backup exit code: =======> 0
2024-10-21 03:50:04,348 - INFO: 2024-10-21 03:50:04,347 - INFO: Exit code: 0
2024-10-21 03:50:04,348 - INFO: 2024-10-21 03:50:04,347 - INFO: Backup files from ===> /backup/
Expand All @@ -62,7 +62,7 @@ crond 4.5 dillon's cron daemon, started with loglevel notice

### Validate database restore

```docker exec -it live-system-mariadb-1 bash```
```docker exec -it live-mariadb-1 bash```

```mysql -uroot -pmy-secret-pw```

Expand All @@ -85,15 +85,15 @@ crond 4.5 dillon's cron daemon, started with loglevel notice

Stop the live system to avoid conflicts as the backup service may try to backup the restored system (The backup and restore services are not intended to run on the same system)

```./stop-live-system.sh```
```./stop-live.sh```

Run the restore

```./restore-backup.sh```
```./start-live-two-with-restore.sh```

Check the restore logs

```docker logs -f restore-test-restore-1```
```docker logs -f live-two-restore-1```

You should see logs similar to:

Expand All @@ -102,28 +102,28 @@ You should see logs similar to:
2024-10-21 03:58:19,579 - INFO: Starting Restore container
2024-10-21 03:58:19,864 - INFO: Restore process container: charming_cohen
2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Status for compose project 'restore-test'
2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Status for compose project 'live-two'
2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Repository: '/restic_data'
2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Restore currently running?: True
2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,074 - INFO: Include project name in restore path?: False
2024-10-21 03:58:20,075 - INFO: 2024-10-21 03:58:20,074 - INFO: Checking docker availability
2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,697 - INFO: ------------------------- Detected Config -------------------------
2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,697 - INFO: service: web
2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,697 - INFO: service: postgres
2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,698 - INFO: Container NAME restore-test-postgres-1
2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME restore-test-postgres-1
2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,698 - INFO: Container NAME live-two-postgres-1
2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME live-two-postgres-1
2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: - postgres (is_ready=True) -> /backup/databases/postgres/test-postgres.sql
2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: service: mariadb
2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME restore-test-mariadb-1
2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME live-two-mariadb-1
2024-10-21 03:58:20,709 - INFO: 2024-10-21 03:58:20,708 - INFO: - mariadb (is_ready=True) -> /backup/databases/mariadb/all_databases.sql
2024-10-21 03:58:20,709 - INFO: 2024-10-21 03:58:20,708 - INFO: -------------------------------------------------------------------
2024-10-21 03:58:20,709 - INFO: 2024-10-21 03:58:20,708 - INFO: Restoring databases
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Restoring up postgres in service postgres
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1
2024-10-21 03:58:20,711 - INFO: 2024-10-21 03:58:20,709 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/postgres/test-postgres.sql | psql --host=restore-test-postgres-1 --port=5432 --username=pguser --dbname=postgres
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1
2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1
2024-10-21 03:58:20,711 - INFO: 2024-10-21 03:58:20,709 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/postgres/test-postgres.sql | psql --host=live-two-postgres-1 --port=5432 --username=pguser --dbname=postgres
2024-10-21 03:58:21,321 - INFO: SET
2024-10-21 03:58:21,321 - INFO: SET
2024-10-21 03:58:21,321 - INFO: SET
Expand Down Expand Up @@ -252,9 +252,9 @@ You should see logs similar to:
2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,387 - INFO: Database Restore exit code: =======> 0
2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,387 - INFO: Exit code: 0
2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,388 - INFO: Restoring up mariadb in service mariadb
2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME restore-test-mariadb-1
2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME restore-test-mariadb-1
2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/mariadb/all_databases.sql | mysql --host=restore-test-mariadb-1 --port=3306 --user=root
2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME live-two-mariadb-1
2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME live-two-mariadb-1
2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/mariadb/all_databases.sql | mysql --host=live-two-mariadb-1 --port=3306 --user=root
2024-10-21 03:58:22,817 - INFO: 2024-10-21 03:58:22,816 - INFO: Database Restore exit code: =======> 0
2024-10-21 03:58:22,817 - INFO: 2024-10-21 03:58:22,817 - INFO: Exit code: 0
2024-10-21 03:58:22,817 - INFO: 2024-10-21 03:58:22,817 - INFO: Restoring files to ===> /restored_data/
Expand All @@ -263,7 +263,7 @@ You should see logs similar to:
```
You can now check the databases for restored data, for example

```docker exec -it restore-test-mariadb-1 bash``
```docker exec -it live-two-mariadb-1 bash``

```mysql -uroot -pmy-secret-pw```

Expand All @@ -285,9 +285,9 @@ You should see logs similar to:
```
### Validate volume restore

We have two services to validate that volumes are restored correctly. see `web` and `web-2` in the `docker-compose-restore-test.yaml` file. When the `docker-compose-restore-service.yaml` file is used the `web` and `web-2` will wait for restore to complete before starting. The service then run ls on the restored mounts to confirm that the restore happened run;
We have two services to validate that volumes are restored correctly. see `web` and `web-2` in the `docker-compose-live-two.yaml` file. When the `docker-compose-restore-service.yaml` file is used the `web` and `web-2` will wait for restore to complete before starting. The service then run ls on the restored mounts to confirm that the restore happened run;

`docker logs -f restore-test-web-1`
`docker logs -f live-two-web-1`

You will see the result

Expand All @@ -301,13 +301,13 @@ tests.py

Stop backup system

```./stop-backup-system.sh```
```./stop-live-two.sh```

## Clean up

Remove Restored data:

```sudo rm -rf ./restored_data```
```sudo rm -rf ./live_two_filestores```

Remove Backup Repo

Expand Down
0 start-live-system.sh → start-live.sh
View file
File renamed without changes.
2 changes: 1 addition & 1 deletion stop-live-two.sh
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
docker compose -p restore-test -f docker-compose-live-two.yaml -f docker-compose-restore-service.yaml down
docker compose -p live-two -f docker-compose-live-two.yaml -f docker-compose-restore-service.yaml down
2 changes: 1 addition & 1 deletion stop-live.sh
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
docker compose -p live-system -f docker-compose-backup-service.yaml -f docker-compose-live.yaml down
docker compose -p live -f docker-compose-backup-service.yaml -f docker-compose-live.yaml down

0 comments on commit 67b8380

Please sign in to comment.