Preserving configs when upgrading from stash 3.x to bitbucket 4.x

[kashyap-parikh]

I have several PR notifier configurations spread over 25+ repositories. When I tried to upgrade from Stash to Bitbucket and upgrade this plugin it failed to carry over those configuration.

Is there way to preserve configs between 3.x to 4.x versions? or copy configs from old to new install to restore?

[tomasbjerre]

There is code for that. That should migrate old settings automatically when upgrading the plugin.

https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket/blob/master/src/main/java/se/bjurr/prnfb/service/SettingsService.java#L253

Can you see anything interesting in the logs when upgrading? Maby a stacktrace? Perhaps search for "looking for legacy settings".

[tomasbjerre]

Should now be fixed in 2.23. You will need to remove the new storage key for 2.18+ se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3. I'm not sure how to do that, perhaps clone production environment to staging environment and try upgrading plugin again.

[kashyap-parikh]

Thank you very much Tomas! I will give this a try on Monday.

[penguingeek]

Thanks Tomas,

I am Joey, I work with @kashyap-parikh. We tried another test run of upgrading to bitbucket 4.6.1 and just an upgrade to plugin v2.23 within stash v3.11.1 - The following are out findings:

_In current stash version 3.11.1 - upgrading to pull-request-notifier-for-stash 1.38 was stable and did not blow away the plugin configs
No option in Stash 3.11.1 to upgrade the plugin beyond pull-request-notifier-for-stash version 1.38

Upgrading from Stash 3.11.1 to bitbucket 4.6.1 blows away pull-request-notifier-for-stash configuration after upgrading to plugin version 2.23

Educated guess is that major upgrade from stash to bitbucket (including mysql schema changes) possibly drops database tables containing plugin data._

Is there a way to extract/export/backup/save pull-request-notifier-for-stash configuration data before upgrading the stash application to bitbucket? If not, what options are available to use to maintain the configuration data over the course of the upgrade?

[tomasbjerre]

I changed the settings format. When the plugin is installed in Bitbucket Server, it will look for legacy settings (se.bjurr.prnfb.admin.AdminFormValues_2) and load them if available and if the new settings (se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3) are not available.

There is an editor, Plugin Data Editor (https://marketplace.atlassian.com/archive/com.atlassian.plugins.plugin-data-editor). But it can only export settings.

I'm not sure how to import settings, I asked a question here:
https://answers.atlassian.com/questions/38849697/plugin-data-editor-remove-settings

[penguingeek]

I have verified that plugin configuration from pull-request-notifier-for-stash is not persisted when upgrading to pull-request-notifier-for-bitbucket v2.23. I attempted to manually extract the admin.vm from the original stash .jar and update the file admin.vm in the bitbucket .jar. this brought over some of the data, but not all of it.

[tomasbjerre]

You may try version 2.17. It has the same GUI and settings format as used in Stash.

Were you able to use the Plugin Data Editor? Is the old key available after upgrade? Is the new key available?

[penguingeek]

After the bitbucket upgrade, there is no option to use an old stash version of the plugin until it has been upgraded.

What is the key for? Can it be migrated from one version to another?

--Joey

Success is the best revenge

On Jun 11, 2016, at 2:44 AM, Tomas Bjerre [email protected] wrote:

You may try version 2.17. It has the same GUI and settings format as used in Stash.

Were you able to use the Plugin Data Editor? Is the old key available after upgrade? Is the new key available?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[tomasbjerre]

You can install 2.17 in Bitbucket Server by downloading it manually from:
https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket/releases/tag/pull-request-notifier-for-bitbucket-2.17

And then uploading it in the admin GUI of Bitbucket Server.

In you case it may be a good idea have 2.17 working. And then as long as the settings key se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3 does not exist, there should be no problem upgrading to latest version of the plugin and having the settings migrated.

[penguingeek]

I can confirm that downgrading to v2.17 does restore the configuration, and that 2.18 is the point where we lose the configuration. Are there any plans to fix this in future releases of this plugin?

[tomasbjerre]

What you need to do is remove the settings key se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3. Have you been able to do that?

[penguingeek]

Where do i remove this key? The only place I was able to find it referenced in the .jar is in se/bjurr/prnfb/service/SettingsService.class which is a binary file.

[tomasbjerre]

It needs to be removed from the bitbucket server database. Im not sure how to do that.

[jmoe-ck]

Ok, so this is where I am at...

I have located 'se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3' in the stash database table stash.key_name. All of the plugin data is located in in the matching 'key_vaule' field. If I remove 'se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3', does this also remove the key_value? ...or are we just removing the the key_name and leaving the rest of that record as is?

[tomasbjerre]

I think you can remove or rename that key. Then the plugin cant find the
key and it will create it again. But im not sure. The plugin code is not
working directly with the database. Just uses a settings api.
Den 24 jun 2016 02:45 skrev "jmoe-ck" [email protected]:

Ok, so this is where I am at...

I have located 'se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3' in
the stash database table stash.key_name. All of the plugin data is located
in in the matching 'key_vaule' field. If I remove
'se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3', does this also
remove the key_value? ...or are we just removing the the key_name and
leaving the rest of that record as is?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#122 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAa1E_oh0uBJLaPyWqXJNgTUA3zGMOsNks5qOyjGgaJpZM4IuLta
.

[jmoe-ck]

After doing a little digging in the stash database, I found the table stash.plugin_setting.

While the plugin_setting table stores plugin-specific configurations, it does such that each plugin has it's own record. The actual plugin configuration data is stored in a single 'key_value' field for it's respective record. Why this matters is that when the 'pull-request-notifier-for-stash' plugin is called for a specific PR notifier, the plugin has to read the entire plugin configuration and parse out that specific trigger's data from every trigger's data every for every trigger call. This is very inefficient and will only get worse over time as more and more triggers are added. A better solution for configurable plugins would be moving configurable plugin data to it's own table, or even more efficient - storing that data in a redis DB. Because the 'pull-request-notifier-for-stash' plugin is unsupported by Atlassian, I don't see this happening anytime in the future. This is not a developer issue, this is an Atlassian issue with the way they handle their plugins. Ultimately this will be an ongoing issue with any configurable plugin.

Also, I don't think your suggestion of deleting/renaming the plugin key would work. Since there is only a single record for each plugin, I don't see how it would generate a new key for our exiting plugin db record, rather than just creating a new fresh plugin record, (essentially leaving our configuration orphaned). I'll test this to confirm, but do not expect it to work.

[jmoe-ck]

Confirmed that removing/renaming the key does not resolve this issue. The data is persistent in the database, but not pulled into the plugin. The same key is regenerated when updating to both 2.18 and the current version 2.26.

Plugin data is persistent in the database and reverting to version 2.17 restores the configuration data back to the plugin UI.

[jmara]

Hey there,

I can confirm the settings behaviour with 2.17 and the latest version :-/ is there any further suggestion to get the settings working after an stash to bitbucket update? Deleting the key in the plugin_settings table has no effect for us either :-(

@kashyap-parikh Do you had any luck so far?

Cheers,
Jan M.

[tomasbjerre]

Its not so hard to create a REST endpoint where a GET request forces the migration to be done. I'll do it when I find the time... And PR:s are welcome! =)

[jmara]

It seems that after removing the old settings key se.bjurr.prnfb.admin.AdminFormValues_2 the settings in the new key are populated and its possible to use the latest version of Bitbucket and your Plugin, we are currently checking all notifications because they are not working out-of-the-box.

[tomasbjerre]

Are you saying that the plugin does not work after upgrade? Or just that the old settings are not migrated?

It the plugin does not work after upgrade, I would need to make that a priority.

[jmara]

It seems that the urlencoding has changed or is no longer used. We are using an internal wrapper to post messages to different slack rooms e.g.:

 https://slackgateway.int/slackroom/chef/?botname=Stash&emoji=stash&msg=*${PULL_REQUEST_USER_DISPLAY_NAME}*+`${PULL_REQUEST_ACTION}` *Pull Req:* ${PULL_REQUEST_TO_REPO_PROJECT_KEY}/${PULL_REQUEST_TO_REPO_NAME} - ${PULL_REQUEST_COMMENT_TEXT} - <https://bitbucket.int/projects/${PULL_REQUEST_TO_REPO_PROJECT_KEY}/repos/${PULL_REQUEST_FROM_REPO_NAME}/pull-requests/${PULL_REQUEST_ID}/overview/${PULL_REQUEST_FROM_BRANCH}> -> ${PULL_REQUEST_TO_BRANCH}

In Stash 3 and the old plugins that seems urlencoded in the latest version we need have to change it to:

https://slackgateway.int/slackroom/chef/?botname=Stash&emoji=stash&msg=*${PULL_REQUEST_USER_DISPLAY_NAME}* %60${PULL_REQUEST_ACTION}%60 *Pull Req:* ${PULL_REQUEST_TO_REPO_PROJECT_KEY}/${PULL_REQUEST_TO_REPO_NAME} - ${PULL_REQUEST_COMMENT_TEXT} - %3Chttps://bitbucket.int/projects/${PULL_REQUEST_TO_REPO_PROJECT_KEY}/repos/${PULL_REQUEST_FROM_REPO_NAME}/pull-requests/${PULL_REQUEST_ID}/overview/%7C${PULL_REQUEST_FROM_BRANCH}%3E -%3E ${PULL_REQUEST_TO_BRANCH}

Except from this, everything is working fine :-)

[tomasbjerre]

If you dont mind, maby you can document how you post to Slack rooms in the readme on same level as the Jenkins headline?

[jmara]

I'll check, the way we post to slack is through an internal company gateway, but I can write a short info that its necessary to urlencode every special character in recent versions

[devalru]

I have an ugly problem. I upgraded from bitbucket 4.x to bitbucket 4.10.0. All fine.
After that I updated/installed the new plugin version 2.37. The plugin is still working but the config
is not displayed.

How can I fix it ? Thank you.

[tomasbjerre]

The original configuration is still applied?
If you add something new, it is not displayed?

[devalru]

"The original configuration is still applied?"
Yes, I see that my jenkins PR jobs are triggered by BitBucket.

"If you add something new, it is not displayed?"
That I did not test because I don't want to enable two parallel triggers.

Is it possible to deactive / disable the old configuration ?

[tomasbjerre]

I wrote some hints on troubleshooting, under "Reporting issues" in the readme. The part about Developer tools is interesting here. Also the part about curl, what is the response of the rest-resources.

Are you a system admin (not only admin)? And we are talking about the global admin page?

[devalru]

Yes, I am sysadmin and have full access. BitBucket Browser plugin (bitbucket/plugins/servlet/restbrowser#/resource/prnfb-admin-1-0-settings/GET) delivers me:

{
"adminRestriction": "SYSTEM_ADMIN",
"shouldAcceptAnyCertificate": false
}

(HTTP 200)
Is the plugin config stored in the mysql database ? If yes in which tables should I look ? Thank you !

P.S.: prnfb-admin-1-0-settings-notifications gives me an JSON-object with my "DEFAULT_PULL_REQUEST_TRIGGER"
with triggers, url, uuid and so on.

But the problem still exist, it is not displayed in the plugin configuration... argh !

[tomasbjerre]

The database is hidden behind API:s used by the plugin. So I dont know
where you will find the config.

There are no exceptions in your logs on machine running Bitbucker Server?

2016-10-19 17:03 GMT+02:00 devalru [email protected]:

Yes, I am sysadmin and have full access. BitBucket Browser plugin
(bitbucket/plugins/servlet/restbrowser#/resource/prnfb-admin-1-0-settings/GET)
delivers me:

{
"adminRestriction": "SYSTEM_ADMIN",
"shouldAcceptAnyCertificate": false
}

(HTTP 200)
Is the plugin config stored in the mysql database ? If yes in which tables
should I look ? Thank you !

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#122 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAa1Ew4yMTygs2dGnsvnIz1HkBxCfovJks5q1jFfgaJpZM4IuLta
.

[devalru]

No exception, but this in the atlassian-bitbucket.log :
127.0.0.1 "GET /plugins/servlet/prnfb/admin HTTP/1.0" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ********* and locales [de_DE, de, en_US, en]

BitBucket debug logging activated.

P.S.: Had plugin version 2.17 installed before.

[tomasbjerre]

There are some logging here:
https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket/blob/master/src/main/java/se/bjurr/prnfb/service/SettingsService.java#L254

You should be able to see some of them, at least from the time when you first browsed to the plugin admin page after upgrade.

[tomasbjerre]

Do you get any notifications when requesting /bitbucket/rest/prnfb-admin/1.0/settings/notifications with GET?

[devalru]

prnfb-admin-1-0-settings-notifications gives me an JSON-object containing my "DEFAULT_PULL_REQUEST_TRIGGER" with trigger-events, url, uuid and so on.

[devalru]

[
{
"headers": [],
"method": "GET",
"name": "DEFAULT_PULL_REQUEST_TRIGGER",
"proxyPort": 8080,
"proxyServer": "myproxy",
"triggerIfCanMerge": "NOT_CONFLICTING",
"triggerIgnoreStateList": [
"MERGED",
"DECLINED"
],
"triggers": [
"OPENED",
"BUTTON_TRIGGER",
"REOPENED",
"RESCOPED_FROM",
"RESCOPED_TO"
],
"url": "myurl",
"uuid": "01c5536c-b73d-4d40-8272-d885f8e0707d"
}
]

[tomasbjerre]

Could your issue be same as: #116 ?

You need to select the notification/button in the dropdown menus.

[devalru]

omg - yes that it is. Thank you very much.

[tomasbjerre]

Only about 10% of the active installations using this plugin are still on 3.x. So I'm closing this one now as I dont see myself or anyone else putting any effort into this issue anymore.