[Jupiter] Debug Garbage Collect Job

[weiweishi]

Describe the bug
We have seen error messages https://rollbar.com/ualbertalib/jupiter/items/369/ regarding the permissions to remove files from era-assets.

We have also experienced very high CPU usage on postgreSQL
https://helpdesk.library.ualberta.ca/otrs/index.pl?Action=AgentTicketZoom;TicketID=47896

that may or may not be related.

Additional things to consider:

• maybe that job needs to watch the clock when it runs?
• limit the number of entries processed?
• self-throttle so as not to eat all the I/O on the DB server?
• better use of config/scheduler.yml, as it runs on all three application servers, and really it only needs to be run once. (adding an env variable as the condition to load the schedule.yml in config/initializer/sidekiq.rb, and let the env variable to be controlled by ansible, to determine which app server is responsible for these scheduled jobs, some ideas: Jobs run 2x as often sidekiq-scheduler/sidekiq-scheduler#88)
• performance test/tune the MySQL query

[weiweishi]

The scheduled cron job will be temporarily stopped before this issue can be fixed - and we can run the manual rake task to remove the orphaned assets.