optionally delete from S3 what was NOT uploaded

[peterbe]

Part of #2224

The reason this PR does not claim to resolve #2224 is partly because 1) this new feature is not enabled yet in prod-build.yml, and 2) let's close the issue once we know it has fully worked in production.

I'm not fond of the name --delete-unrecognized. But writer's block is real so I plowed through with it. --delete-old is equally bad (or worse) because time isn't important (exception for static assets).
What about --delete-leftovers? Or --delete-remaining? Or --delete-not-uploaded?

I've been testing this locally with my personal S3 bucket which is only the en-US stuff. I have been uploading to that for testing the Deployer dating back to mid-2020. So my S3 bucket was full of all sorts of junk:

▶ DEPLOYER_LOG_EACH_SUCCESSFUL_UPLOAD=false poetry run deployer  upload --content-root=/Users/peterbe/dev/MOZILLA/MDN/content/files --bucket=peterbe-yari --delete-unrecognized /tmp/build
Deployer (0.3.0)
Upload files from: /tmp/build
Upload redirects from: /Users/peterbe/dev/MOZILLA/MDN/content/files
Upload into: main/ of peterbe-yari
Total pending redirect uploads: 12,197 (10.2ms)
Total pending file uploads: 48,280 (1.4s)
Total existing S3 objects: 77,407 (21.7s)
  [▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋]  60477/60477  100%
Total uploaded files: 0 (0B)
Total uploaded redirects: 12,197
Total skipped files: 48,280 matched existing S3 objects
Total upload/skip time: 34.6s
Total pending task deletions: 16,886
  [▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋]  16886/16886  100%
Total deleted keys: 16,886
Done in 1m28s.

And if I run it a second time, immediately after I get:

▶ DEPLOYER_LOG_EACH_SUCCESSFUL_UPLOAD=false poetry run deployer upload --content-root=/Users/peterbe/dev/MOZILLA/MDN/content/files --bucket=peterbe-yari --delete-unrecognized /tmp/build
Deployer (0.3.0)
Upload files from: /tmp/build
Upload redirects from: /Users/peterbe/dev/MOZILLA/MDN/content/files
Upload into: main/ of peterbe-yari
Total pending redirect uploads: 12,197 (9.9ms)
Total pending file uploads: 48,280 (1.4s)
Total existing S3 objects: 60,521 (19.0s)
  [▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋]  60477/60477  100%
Total uploaded files: 0 (0B)
Total uploaded redirects: 12,197
Total skipped files: 48,280 matched existing S3 objects
Total upload/skip time: 35.8s
Total pending task deletions: 0
Total deleted keys: 0
Done in 56.2s.

Here, note the lines:

Total pending task deletions: 0
Total deleted keys: 0

[peterbe]

To double-check, I also manually uploaded some random file from my desktop into some random folder:

Run the deployer again:

Upload files from: /tmp/build
Upload redirects from: /Users/peterbe/dev/MOZILLA/MDN/content/files
Upload into: main/ of peterbe-yari
Total pending redirect uploads: 12,197 (13.1ms)
Total pending file uploads: 48,280 (2.1s)
Total existing S3 objects: 60,522 (19.6s)
  [▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋]  60477/60477  100%
Total uploaded files: 0 (0B)
Total uploaded redirects: 12,197
Total skipped files: 48,280 matched existing S3 objects
Total upload/skip time: 37.4s
Total pending task deletions: 1
Total deleted keys: 1
Done in 59.2s.

And refresh in the AWS S3 console UI:

[escattone]

@peterbe Overall, nicely done, but I do have some comment nits, some suggested changes, and one bug fix.

[escattone]

Awesome, thanks @peterbe!