Jupiter UAT – What's Broken, What Needs Fixing

[mbarnett]

Specific questions I've thought of:

• Is it working at all?
• Are queues pumping?
• Are file uploads working?
• What data is in UAT (synthetic stuff from the seeds, or an old dump?)
• Is it still the Fedora version
• Should we blow the data away and start over with post-Fedora, or try upgrading it?

[pgwillia]

Specific questions I've thought of:

* Is it working at all?

Yes, most of it is working.

• search
• download
• view
• deposit - sort of
• admin

* Are queues pumping?

No. This deposit from three months ago is still enqued http://uat.library.ualberta.ca/items/b78fbf20-9519-4b31-a455-df7a6475d542

* Are file uploads working?

Not exactly. The wizard show's the upload,

sidekiq shows the job in the queue

But they don't finish processing

* What data is in UAT (synthetic stuff from the seeds, or an old dump?)

The data is mostly seeds but there are three 'user' deposited items including the one I created

* Is it still the Fedora version

This is the Fedora version <meta name="generator" content="Jupiter 1.2.17 - https://github.com/ualbertalib/jupiter">

* Should we blow the data away and start over with post-Fedora, or try upgrading it?

The data is mostly seeds with a few user created entries. I don't think there would be much value in trying to migrate it other than the exercise of migration itself -- which is being done for real in staging by other members of the team.

[pgwillia]

I looked at watchtower for keeping our deployment up to date

With watchtower you can update the running version of your containerized app simply by pushing a new image to the Docker Hub or your own image registry. Watchtower will pull down your new image, gracefully shut down your existing container and restart it with the same options that were used when it was deployed initially.

It can even be used in a docker-compose.yml

version: "3"
services:
  cavo:
    image: index.docker.io/<org>/<image>:<tag>
    ports:
      - "443:3443"
      - "80:3080"
  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json
    command: --interval 30

One problem with this approach might be

By default, watchtower will watch all containers. However, sometimes only some containers should be updated.
but there is a workaround to include or exclude using a label

An alternative to this which I used on the discovery deployment is having a cron job update

[deploy@uatsrv01 ~]$ crontab -l
@daily cd /home/deploy/discovery && docker-compose pull && docker-compose up -d

This has been relatively successful Discovery 3.2.2 is the current version and the current version deployed.

[pgwillia]

These were the roadblocks for UAT we would need to address to move forward with this plan from the ERA Software Development Process Ideas:

• UAT server: for Jupiter this has been sitting idle for a while. We'll need to get it back in shape.
  • We also need to figure out the data situation on UAT – it’s not the most useful environment if its contents are purely synthetic and don’t regularly experience or show issues in a way that Ticket Owners can see when something is fixed…
    • But we don’t have automated mechanisms for cloning data back from Production, and we likely will not any time soon.
• We can't (easily) automate a push to UAT from GitHub/Continuous Integration right now, due to firewall rules and VLAN/test infrastructure restrictions imposed by Ops.
  • Rather than trying to get this opened up to allow incoming webhooks from CI, which is an argument we lost last time around, we could look at just putting a cron job or similar on the local jenkins server (or in the worst case, my otherwise-unused workstation) to regularly check for and deploy new containers automatically.

And @jefferya said,

Some additional pieces I’ve found useful for a UAT environment:

• Https: I’ve sometimes run across instances where a “thing” works in http but fails under https
• Distinct server names or routes to applications instead of port ‘x’ is Jupiter, port ‘y’ is avalon, etc. Personally, I find names easier to remember than numbers
• If the applications are containerised, a reverse proxy could handle the routing and Let’s Encrypt certs (for example Traefik or Envoy.

[pgwillia]

• update https://github.com/ualbertalib/di_internal/blob/6fce8df94b80d7ff215efd244e479e00d7b20314/System-Adminstration/UAT-Environment.md#deploy-jupiter
• update https://github.com/ualbertalib/jupiter#for-production-deployment-on-uat-environment
• update https://github.com/ualbertalib/jupiter/blob/integration_postmigration/bin/predeploy
• solve permission issue on uat (root owns a number of directories including solr)

[pgwillia]

Used ./deploy. It cleaned up a number of dangling images and updated to version 1.2.20

[pgwillia]

curl -O https://raw.githubusercontent.com/ualbertalib/jupiter/master/bin/predeploy
chmod u+x predeploy
./predeploy
cd jupiter/
chmod u+x deploy
./deploy
docker-compose exec web bash
> rake db:seed

cd ..
curl -O https://raw.githubusercontent.com/ualbertalib/jupiter/integration_migration_phase1/bin/predeploy
./predeploy
cd jupiter 
./deploy
docker-compose exec web bash
> rake jupiter:get_me_off_of_fedora

irb(main):006:0> ArItem.count
D, [2020-07-17T22:02:11.058200 #110] DEBUG -- :    (2.1ms)  SELECT COUNT(*) FROM "ar_items"
=> 130
irb(main):007:0> ArThesis.count
D, [2020-07-17T22:02:15.539619 #110] DEBUG -- :    (2.1ms)  SELECT COUNT(*) FROM "ar_theses"
=> 100
irb(main):009:0> ArCommunity.count
D, [2020-07-17T22:04:12.742193 #110] DEBUG -- :    (2.6ms)  SELECT COUNT(*) FROM "ar_communities"
=> 18
irb(main):010:0> ArCollection.count
D, [2020-07-17T22:04:18.800864 #110] DEBUG -- :    (1.9ms)  SELECT COUNT(*) FROM "ar_collections"
=> 22

cd ..
curl -O https://raw.githubusercontent.com/ualbertalib/jupiter/integration_postmigration/bin/predeploy
./predeploy
cd jupiter 
# edited docker-compose.yml to use the image I built tagged ignore_cache 
# edit .env_deployment RACK_ATTACK_SAFELISTED_IPS=""
./deploy 

irb(main):001:0> Community.find_each(&:save!)
irb(main):003:0> Collection.find_each(&:save!)