Upgrading from v1.x to 2.0.0

Upgrading from v1.x to 2.0.0

The 2.0.0 release represents a major upgrade to the Roadmap codebase. One important aspect of these changes involves an emphasis on data validation. Prior iterations of the Roadmap system did not have robust data validation and it is likely that data imported from DMPonline_v3 or other systems introduced invalid records.

These new validations will cause invalid records to fail when accessed or updated by your users so it is imperative that you follow the instructions below to ensure that your site functions properly after the upgrade.

We highly recommend that you backup your existing database before running through these steps to prepare your system for Roadmap 2.0.0!

Overview of Database changes

• Added new table stats to facilitate the Usage Statistics page available to Admins
• Added new versionable_id field to the following tables to help ensure accurate transfer of template customizations when a funder template changes: annotations, questions, sections, phases
• Added default: false to boolean fields: guidance_groups.published, guidance_groups.optional_subset, notes.archived, orgs.is_other
• Added limit: 80 to users.email
• Dropped unused tables: file_types, file_uploads, splash_logs
• Removed unused fields: guidances.question_id, orgs.wayfless_entity, orgs.parent_id, orgs.banner_text, orgs.logo_file_name, phases.slug, plans.slug, users.orcid_id, users.shibboleth_id
• Various index changes in support of the above changes

Overview of model validations

Many new validations were introduced to enforce referential integrity, data types, etc. The following list contains highlights of some of these validations. Please review the validation section of each of the models for a complete list of changes.

• Annotation type must be unique per question (currently only 2 options: example answer or guidance)
• Guidance records must now have at least one associated theme if they are published.
• Language abbreviations must follow the ISO standard using dashes instead of underscores (e.g. pt-BR not pt_BR)
• Org now requires a contact_name, contact_email, feedback_msg and feedback_subject when feedback_enabled is true
• Guidance and Templates can no longer be published if they belong to the 'Other' Org, orgs.is_other = true
• User must be associated with an Org
• Templates must now have a family_id
• User must have a firstname and surname
• Question, Annotation and Note text cannot be blank
• Org names must now be unique and they must have an abbreviation
• Template, Phase, Section, Plan and Theme title cannot be blank
• Phase, Section and Question numbers must be unique for their given context (e.g. Section #23 cannot have 2 question #1s)

Database upgrade

You will need to run through the following steps to upgrade your system to v2.0.0 and ensure that your existing database records are valid:

Part One

Please run rake upgrade:v2_0_0_part_1

The Part one script runs 3 separate steps to prepare your system for Roadmap v2.0.0:

1. Updates existing records in your database to prepare them for some new database validations that prevent NULL values in boolean fields:

• guidance_groups.optional_subset - any NULL values will be set to 'false'
• guidance_groups.published - any NULL values will be set to 'false'
• notes.archived - any NULL values will be set to 'false'
• orgs.is_other - any NULL values will be set to 'false'

2. The standard Rails database migrations will then be run
3. A script that checks your existing data to determine if any records would become invalid due to the new validations added to each of the models. This script runs SQL queries to do a quick scan of your data. It is intended to give you an idea of the extent of any issues you may have before running the more in depth scripts in part two.

The final script will output a report to your console. That looks something like this:

In the example below, we can see that there are 36 orphaned records in my answers table and 3 user records that do not have a unique email address:

Checking Annotation records
  0 records with a empty text field
  0 orphaned records due to nil or missing org
  0 orphaned records due to nil or missing question
  0 records with a empty type field
  0 records that are not unique per (type, question_id, org_id)
Checking Answer records
  0 orphaned records due to nil or missing plan
  0 orphaned records due to nil or missing user
  36 orphaned records due to nil or missing question
  0 records that are not unique per (question_id, plan_id)

...

Checking User records
  0 records with a empty email field
  3 records that are not unique per (email)

Part two

If you were fortunate enough to only find a few invalid records in the report from part one, you can go ahead and clean them up manually. We advise documenting any queries you run though in the event that you need to run them again on another instance of your database.

Finding invalid records

Please run the following script to run a thorough investigation of every record in your database. (Note that this can take in excess of one hour depending on the size of your database):

rake data_cleanup:find_invalid_records

This script will output another report detailing any additional records that are still invalid

Correct any data issues

Review the report from the prior rake task and then either correct the issues manually or create your own data cleanup rules that can then be run through the rake data_cleanup:clean_invalid_records task. This task will pick up any rules you have added to the lib/data_cleanup/rules directory.

Please watch the following video for a detailed overview of this process: DMPRoadmap v2.0.0 DataCleanup

Iterate through the find/clean steps until you can run rake data_cleanup:find_invalid_records and verify that all of your existing data is valid.

Part Three

The final script will perform a few other tasks to initialize some of the new tables and fields.

1. Runs a script to initialize all of the new versionable_id fields that were added to the phases, sections, questions and annotations tables.
2. Normalizes your languages abbreviations
3. Populates the new usage statistics tables

Please note that the task that updates the versionable_id fields can take in excess of 15 minutes depending on the size of your database.

Updating your deployment scripts

This release also involved some changes to the basic infrastructure of the system. Please check and update your deployment scripts as needed for your individual installations.

• Tests are now run via RSpec instead of MiniTest. Use rpec spec/ to run the tests
• Modifications were made to the Webpack configuration. To compile you assets for production (or any situation where you do not want Webpack to watch your assets for changes) you should now use npm run bundle -- -p --no-watch
• We have updated the Gemfile to segregate deployment specific gems so that you can exclude them as needed when running bundle install --without [groupA] [groupB]. For example: bundle install --without mysql puma thin:
  • puma
  • thin
  • mysql
  • postgres
• You will need to setup a job (e.g. via CRON) that runs the rake stat:build_monthly task at the beginning of each month