Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

notary-server and notary-signer cannot start after upgrading to 1.7.4 #7091

Closed
reasonerjt opened this issue Mar 7, 2019 · 6 comments
Closed

Comments

@reasonerjt
Copy link
Contributor

reasonerjt commented Mar 7, 2019

the log of notary-server container:

Mar  7 06:52:25 172.18.0.1 notary-server[5654]: 2019/03/07 06:52:25 Updating database.
Mar  7 06:52:25 172.18.0.1 notary-server[5654]: 2019/03/07 06:52:25 Done updating database.
Mar  7 06:52:25 172.18.0.1 notary-server[5654]: error: file does not exist
Mar  7 06:52:25 172.18.0.1 notary-server[5654]: waiting for postgres://server:password@postgresql:5432/notaryserver?sslmode=disable to come up.
Mar  7 06:52:26 172.18.0.1 notary-server[5654]: error: file does not exist
Mar  7 06:52:26 172.18.0.1 notary-server[5654]: waiting for postgres://server:password@postgresql:5432/notaryserver?sslmode=disable to come up. 

The file does not exist error was due to an inconsistecy introduced in migration that switch Harbor DB from mysql/mariadb to postgresql

There's no good way to fix it easily, a quick fix is to drop all tables of the notaryserver and notarysigner databases, and re-sign the images that were signed previously.
A better workaround involve some manual work is in progress.

@RRAlex
Copy link

RRAlex commented Mar 7, 2019

Upgraded from 1.4.0 -> 1.6.3 without issue, was about to upgrade to 1.7.4 but saw the warning...
Does the bug manifests itself with previous 1.7.x?
Or is upgrading to 1.7.3, until 1.7.5 arrives, a safe move?
Cheers!

wy65701436 added a commit to wy65701436/harbor that referenced this issue Mar 8, 2019
The script is to fix the issue goharbor#7091, notary upgrade failure from mysql to pqsl.

Signed-off-by: wang yan <[email protected]>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Mar 8, 2019
The script is to fix the issue goharbor#7091, notary upgrade failure from mysql to pqsl.

Signed-off-by: wang yan <[email protected]>
@wy65701436
Copy link
Contributor

@RRAlex it happens on v1.7.4 forward, and we will provide a manual workaround, you have to run a script at your end to fix it.

wy65701436 added a commit to wy65701436/harbor that referenced this issue Mar 8, 2019
The script is to fix the issue goharbor#7091, notary upgrade failure from mysql to pqsl.

Signed-off-by: wang yan <[email protected]>
@wy65701436
Copy link
Contributor

Just run the script(https://github.com/goharbor/harbor/blob/master/tools/notary-migration-fix.sh) to fix notary DB before launching Harbor.

@madic-creates
Copy link

madic-creates commented Jun 18, 2019

I've upgraded from 1.6.0 to 1.8.1 and came accross the same issue and did run the notary-migration-fix.sh script but still get the "file does not exist" error. Any idea?

root@harbor:/opt/harbor# docker images goharbor/harbor-db --format "{{.Repository}}:{{.Tag}}"
goharbor/harbor-db:v1.8.1
goharbor/harbor-db:v1.6.0

When running the script it uses the goharbor/harbor-db:v1.6.0 instead of v1.8.1 image. Is this correct?

The script also stops with the message "Harbor DB cannot reach within one minute."

@apallier
Copy link

Hi, I have the same issue and same behavior as @Madic- when I run the script on a 1.8.2 upgrade.
Should I run this script when Harbor is running or stopped ?

@madic-creates
Copy link

"Fixed" the problem by dropping and then re-creating the databases notaryserver and notarysigner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants