mc mirror testing

Connect to source/client

ssh -p 20431 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Connect to target

ssh -p 20673 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"

Installing certs

rm -rf certgen-linux-amd64
wget http://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64
chmod +x certgen-linux-amd64
./certgen-linux-amd64 -host "127.0.0.1"
mkdir -p ~/.minio/certs/CAs
cp public.crt ~/.minio/certs
cp private.key ~/.minio/certs

Run minio on both

loginctl enable-linger ubuntu
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
mkdir data
./minio server data --address :9000 --console-address :9090 --certs-dir ~/.minio/certs & >> /tmp/minio.log

Install mc on source

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin

On source/client create aliases

mc alias set source https://mc-mirror-source.lab.min.dev:9000 minioadmin minioadmin
mc alias set target https://mc-mirror-target.lab.min.dev:9000 minioadmin minioadmin

Create files

mkdir source
cd source
dd if=/dev/urandom bs=1M count=128 iflag=fullblock of=object
cd ~
mkdir -p ~/raw/prefix
cd ~/raw
split -b 1MiB --additional-suffix=.ext ~/source/object object
split -b 1MiB --additional-suffix=.jpg ~/source/object object
cd ~/raw/prefix
split -b 1MiB --additional-suffix=.pdf ~/source/object object
split -b 1MiB --additional-suffix=.doc ~/source/object object
split -b 1MiB --additional-suffix=.txt ~/source/object object

Check file sizes

ls -1 ~/raw | wc -l
ls -1 ~/raw/prefix | wc -l
#### 257 (-1)
#### 384
#### = 640

Unversioned

Buckets setup

mc ls source/
mc rm source/unversioned
mc mb source/unversioned

Trigger mirror fs to s3

mc mirror --watch --overwrite --preserve  ~/raw/ source/unversioned/raw/
mc cp --recursive ~/raw/ source/unversioned/raw

Trigger mirror s3 to s3

mc mb target/unversioned
mc ls source/unversioned
mc mirror --watch --overwrite --preserve source/unversioned/raw/ target/unversioned/raw/

Validate all 640 files transferred

Validate all extensions respected

mc ls --recursive --summarize target/
#### Total Size: 640 MiB
#### Total Objects: 640

Create new files on source

cd ~/raw/prefix
split -b 2MiB --additional-suffix=.pdf ~/source/object object
split -b 3MiB --additional-suffix=.doc ~/source/object object
split -b 4MiB --additional-suffix=.txt ~/source/object object

Check file sizes

ls -1 ~/raw | wc -l
ls -1 ~/raw/prefix | wc -l
#### 257 (-1)
#### 384
#### = 640

Validate all 640 files transferred

Validate all extensions respected

mc ls --recursive --summarize target/
#### Total Size: 885 MiB
#### Total Objects: 640

Purge source prefix

mc rm --recursive --force source/unversioned/raw/prefix

Validate target is purged

mc ls --recursive --summarize source/unversioned/raw/prefix
#### Total Size: 0 B
#### Total Objects: 0
mc ls --recursive --summarize target/unversioned/raw/prefix
#### Total Size: 0 B
#### Total Objects: 0

Validate target still contains unpurged data

mc ls --recursive --summarize target/
#### Total Size: 256 MiB
#### Total Objects: 256

Validate prefix trimming of prefix/

mc tree --files source/unversioned | grep prefix
mc tree --files target/unversioned | grep prefix

--

Versioned

mc ls source/
mc rb --force source/versioned
mc mb source/versioned
mc rb --force target/versioned
mc mb target/versioned
mc version enable source/versioned
mc version enable target/versioned

Add mirrors

mkdir -p raw_versioned/prefix
rm -rf ~/raw_versioned/*
mc mirror --watch --overwrite --preserve  ~/raw_versioned source/versioned/raw
mc mirror --watch --overwrite --preserve source/versioned/raw target/versioned

Copy data to fs - create three versions

cp -R ~/raw ~/raw_versioned
cp -R ~/raw ~/raw_versioned
cp -R ~/raw ~/raw_versioned

FLAG - second trigger s3s3 will not take effect. It must be recreated. Data is intact FLAG - even though all copying is finished on fs, versions will be missing on f3s3 mirror; thus s3s3 is also broken FLAG - everything below is happy path only

Validate all 640 files transferred

Validate all extensions respected

mc ls --recursive --summarize source/versioned/raw
#### Total Size: 885 MiB
#### Total Objects: 640
mc ls --recursive --summarize --versions source/versioned/raw
#### Total Size: 2.6 GiB
#### Total Objects: 1920

Validate target get single version

mc ls --recursive --summarize target/versioned/raw
#### Total Size: 885 MiB
#### Total Objects: 640
mc ls --recursive --summarize --versions target/versioned/raw
#### Total Size: 885 MiB
#### Total Objects: 640

Keep note of prefix/

mc ls --recursive --summarize source/versioned/raw/raw/prefix
Total Size: 629 MiB
Total Objects: 384
mc ls --recursive --summarize --versions source/versioned/raw/raw/prefix
Total Size: 1.8 GiB
Total Objects: 1152

Delete data in prefix (create delete marker)

mc rm --recursive --force source/versioned/raw/raw/prefix

Validate all 384 versions deleted and 384*3=1152 files remain

Validate all extensions respected

mc ls --recursive --summarize source/versioned/raw/raw/prefix
Total Size: 0 B
Total Objects: 0
mc ls --recursive --summarize --versions source/versioned/raw/raw/prefix
Total Size: 1.8 GiB
Total Objects: 1536

Validate target get single version - delete markers are not copied

mc ls --recursive --summarize target/versioned/raw/prefix
Total Size: 629 MiB
Total Objects: 384
mc ls --recursive --summarize --versions target/versioned/raw/prefix
Total Size: 629 MiB
Total Objects: 

Test json output

--json flag creates a json stream which must be .. streamed