Migrating to 1.9

[ncstc1]

Hello there,

We are trying to upgrade from 1.7.1 to 1.9.0.

The result of the migration seems ok:

db.versions.find()
{
  _id: '1.7.1',
  initialInstall: 1708090032.24
}
{
  _id: '1.7.5',
  migrationStarted: 1708090032.247,
  migrationCompleted: 1708090032.251
}
{
  _id: '1.8.0',
  migrationStarted: 1708090032.252,
  migrationCompleted: 1708090033.201
}
{
  _id: '1.9.0',
  migrationStarted: 1708090033.202,
  migrationCompleted: 1708090033.22
}

Now it looks like users are not available anymore.

For example:

• the admin/getUsers (GET {register}/admin/users) returns an empty array
• the previously working login information now fails (Unknown user)

Do you see what the problem could be? (the migrations to 1.9 clearly deletes the system/username events and seems to move them to an sqlite DB but those DBs seem empty in our case).

Is there an additional migration step for this version?

Thanks,
Stéphane.

[perki]

Hello Stéphane,

You're mentioning a migration of open-pryv.io right? Not an enterprise version?

Pierre-Mikael

[ncstc1]

Yes, exactly, not an enterprise version

[perki]

Hello Stéphane,

Using a sample dataset produced with open-pryv.io 1.7.1 which can be found here we actually found 2 issues.

1. the migration was not triggered as the version tag was not present.
2. the migration of attachments was failing as the folder name changed from attachment-files to attachments

What is surprising from your initial post is that you are getting some result from "db.versions.find()" which we don't. This means that the issue you're facing might come from a different cause. Please check the following steps to migrate from v1.7.1 to v1.9.0 and let us know if it solves your problem.

If you're running a dockerized version, make a backup with docker exec -t open-pryv-mongo mongodump -d pryv-node -o /data/backup/. The migration has to be done on a native set-up, but the migrated data will be usable by a docker installation.

1. Make a dump a of v1.7.1 the database with ./var-pryv/mongodb-bin/bin/mongorestore <db_backup_destination>
2. Prepare a fresh set-up of v1.9.0 and copy the following directories into v1.9.0 var-pryv dir
   • {v1.7.1}/var-pryv/attachement-files => {v1.9.0}/var-pryv/attachements
   • {v1.7.1}/var-pryv/users => {v1.9.0}/var-pryv/users
3. Start mongodb for v1.9.0
   • Restore data from the dump folder ./var-pryv/mongodb-bin/bin/mongorestore <db_backup_destination>
   • Connect to mongodb with {v1.9.0}/var-pryv/mongodb-bin/bin/mongosh
   • From mongoshell type use pryv-node to connect to pryv.io db.
   • Then check the current version with: db.versions.find({}) if empty add one with: db.versions.insertOne({ _id: '1.7.1', initialInstall: 0 })
4. Run the migration script: LOGS=info NODE_ENV=production ./components/api-server/bin/migrate --config {your configuration file}
   At this stage you see the migration process and user correctly migrated should be listed at step 1.8.0 and attachments (if any) at step 1.9.0

Here is a sample output

... some warnings
2024-02-23 15:51:48 [info]: [api-server:migration-process] Connecting to storage... 
2024-02-23 15:51:48 [info]: [api-server:migration-process] Storage connected. 
2024-02-23 15:51:48 [info]: [api-server:migration-process] Starting migration... 
2024-02-23 15:51:48 [info]: [api-server:migration-1.7.5] V1.7.1 => v1.7.5 Migration started 
2024-02-23 15:51:48 [info]: [api-server:migration-1.7.5] V1.7.1 => v1.7.5 Migration finished 
2024-02-23 15:51:48 [info]: [api-server:migration-1.8.0] V1.7.5 => v1.8.0 Migration started 
2024-02-23 15:51:48 [info]: [api-server:migration-1.8.0] Migrating userId: clsyq5bum00001q9ylf5d2exx username: testuser 
2024-02-23 15:51:48 [info]: [api-server:migration-1.8.0] Migrating password for userId: clsyq5bum00001q9ylf5d2exx 
2024-02-23 15:51:48 [info]: [api-server:migration-1.8.0]  migrating 5 fields for userId: clsyq5bum00001q9ylf5d2exx 
2024-02-23 15:51:48 [info]: [api-server:migration-1.8.0] V1.7.5 => v1.8.0 Migration finished 
2024-02-23 15:51:48 [info]: [api-server:migration-1.9.0] V1.9.0 => v1.9.0 Migration started 
2024-02-23 15:51:48 [info]: [api-server:migration-1.9.0:attachments] Attachmend moved for userId: clsyq5bum00001q9ylf5d2exx to: /Users/perki/code/open-pryv.io/var-pryv/users/x/x/e/clsyq5bum00001q9ylf5d2exx/attachments 
2024-02-23 15:51:48 [info]: [api-server:migration-1.9.0] V1.8.0 => v1.9.0 Migration finished 
2024-02-23 15:51:48 [info]: [api-server:migration-process] Storage migration complete. 
2024-02-23 15:51:48 [info]: [api-server:migration-process] Process exiting. 

[ncstc1]

This is perfect: I could run the upgrade process with the instructions above.

Now users can login into a version 1.9 deployment.

The key point for us was to define a new version document if the collection was empty.

Thank you,
Stéphane

[perki]

Thanks for the feedback,
We keep track of the issue solution on https://github.com/orgs/pryv/discussions/3