[QA] Update to 10.11.0 kills primary S3 storage #582

jnweiger · 2022-07-13T15:37:18Z

Seen with files_primary-2.0.0-rc.1 and owncloud core 10.11.0-alpha.1

files_primary_s3 2.0.0-rc.1 cannot be installed on 10.10.0 - OK (intended min-version=10.11)

occ app:enable files_primary_s3
In app.php line 1129:
  App "S3 Primary Object Storage" cannot be installed because the following dependencies are not fulfilled: ownCloud 10.11 or higher is required.

files_primary_s3 2.0.0-rc.1 runs fine on 10.11.0-alpha.1 - OK
files_primary_s3 1.2.0 runs fine on 10.10.0. - OK
files_primary_s3 1.2.0 explodes on 10.11.0-alpha.1 with internal server error. - BAD
( both fresh install 10.11.0-alpha.1 and update from 10.10.0 to 10.11.0 - update details see below )

Web Interface cannot log in. Internal server error. The log has:

"reqId":"Ys4hm5gO6k5od@IOmJ7VsAAAAAQ","level":3,"time":"2022-07-13T01:36:27+00:00","remoteAddr":"80.136.152.232","user":"admin","app":"index","method":"POST","url":"\/index.php\/login","message":"Exception: {\"Exception\":\"Error\",\"Message\":\"Class 'GuzzleHttp\\\\Ring\\\\Client\\\\StreamHandler' not found\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/apps-external\\\/files_primary_s3\\\/lib\\\/s3storage.php(124): OCA\\\\Files_Primary_S3\\\\S3Storage->init()\\n#1 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/ObjectStoreStorage.php(456): OCA\\\\Files_Primary_S3\\\\S3Storage->writeObject()\\n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(360): OC\\\\Files\\\\ObjectStore\\\\ObjectStoreStorage->touch()\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Availability.php(372): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->touch()\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(360): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Availability->touch()\\n#5 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(360): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->touch()\\n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(360): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->touch()\\n#7 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/View.php(1226): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->touch()\\n#8 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/View.php(602): OC\\\\Files\\\\View->basicOperation()\\n#9 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Node\\\/Folder.php(179): OC\\\\Files\\\\View->touch()\\n#10 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/legacy\\\/util.php(429): OC\\\\Files\\\\Node\\\\Folder->newFile()\\n#11 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/legacy\\\/util.php(423): OC_Util::copyr()\\n#12 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/legacy\\\/util.php(400): OC_Util::copyr()\\n#13 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/User\\\/Session.php(465): OC_Util::copySkeleton()\\n#14 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/User\\\/Session.php(553): OC\\\\User\\\\Session->prepareUserLogin()\\n#15 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/User\\\/Session.php(340): OC\\\\User\\\\Session->loginWithPassword(*** sensitive parameters replaced ***)\\n#16 \\\/var\\\/www\\\/owncloud\\\/core\\\/Controller\\\/LoginController.php(245): OC\\\\User\\\\Session->login(*** sensitive parameters replaced ***)\\n#17 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(170): OC\\\\Core\\\\Controller\\\\LoginController->tryLogin(*** sensitive parameters replaced ***)\\n#18 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(89): OC\\\\AppFramework\\\\Http\\\\Dispatcher->executeController()\\n#19 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/App.php(100): OC\\\\AppFramework\\\\Http\\\\Dispatcher->dispatch()\\n#20 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Routing\\\/RouteActionHandler.php(47): OC\\\\AppFramework\\\\App::main()\\n#21 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Route\\\/Router.php(344): OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler->__invoke()\\n#22 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(914): OC\\\\Route\\\\Router->match()\\n#23 \\\/var\\\/www\\\/owncloud\\\/index.php(54): OC::handleRequest()\\n#24 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/apps-external\\\/files_primary_s3\\\/lib\\\/s3storage.php\",\"Line\":75}"}

This will happen to all installations using files_primary_s3, when we push 10.11.0 to the online updater:

online updater announces availability of 10.11.0
admin reacts and updates core
files_primary_s3 is not updated, as it is not bundled. - Okayish
files_primary_s3 not updated to version 2.0.0 when that is available from marketplace. BAD.
ownCloud no longer works after the update: Internal server error. - BAD
The app cannot be disabled, as it manages the primary storage for all users. - BAD

Background:

guzzle has a major version upgrade and is incompatible to its previous
version. Both core and apps must be upgraded simultaneously to avoid this incompatibility.
Update guzzle major version from 5 to 7 core#39387
The online updater pulls only the minimal tar ball.
files_primary_s3 is also not bundled in the complete tar ball.

Expected behaviour

When owncloud core 10.11.0 detects the incompatible files_primary_s3, a useful error message is printed
'This version of ownCloud requires files_primary_s3 version 2.0.0 - please upgrade this app.'

Manual workaround:

After updating core to 10.11.0, admin downloads a 2.x.x tar from https://github.com/owncloud/files_primary_s3/releases
and unpacks it manually into the apps_external folder
Then owncloud should be operational again.

Improvement ideas:

Can we add a migration step to 10.11.0 that checks the version of files_primary_s3 and either warns or triggers the app update
automatially. This could be easily scaled to include more apps with the same problem.
(Alternatively, provide a script that upgrades core and affected apps in one step)
Can we add files_primary_s3 to the complete bundle? (does not directly help with the updater)
Can we push the complete bundle via online updater?
All current apps have max-version=10 - this could be used as a technical solution, if we call the upcoming server release owncloud 11.0.0 instead of 10.11.0
Downside: all apps must be re-released with max-version=11. (My least favorite idea.)

The text was updated successfully, but these errors were encountered:

jnweiger · 2022-07-13T15:40:35Z

@phil-davis @pmaier1 do you have suggestions how to handle this?
It is the first app where I examined the guzzle situation. There may be more apps with a similar problem.
For most apps I'd expect, that an admin can disable an incompatible app during upgrade, to make owncloud work again. Then use the Marketplace to get the new app version.

In either case, we should communicate the situation and the available options well.

phil-davis · 2022-07-14T05:47:05Z

if we call the upcoming server release owncloud 11.0.0 instead of 10.11.0

Bit of a brain dump:

This bit really needs thinking about. Quite a few apps actually have direct calls into the Guzzle library and code had to be edited. The existing versions of the apps have core max-version 10 - they claim that they will continue to "work" with any future version 10.*.*. The semver "promise" is that all core releases numbered 10.*.* will continue to have upwardly-compatible "public interfaces" - that means real API endpoint behavior, but also that any core classes/methods that are built-in to core and "advertised" as ways for apps to "do stuff" are also upwardly-compatible. The usual examples are the various hooks for file events that apps can subscribe and listen to, ways to log stuff, ways to find out about the current user, group and so on. But this Guzzle case is more unusual. Both core and apps have made direct use of the Guzzle5 interface to do "client request" things, and Guzzle6 and Guzzle7 have changed that interface quite a bit. It would be easier if core had provided a set of wrapper classes/methods around Guzzle, and all apps used those - then we could try to keep those wrapper classes upwardly compatible, and inside those classes they could start using Guzzle7 without apps even realizing.

But I think it is too late for that now - there are apps in-the-wild that claim to work with all future 10.*.* but clearly will not work with the current core master (which will become 10.11.0 or 11.0.0 or whatever release numbering)

Technically, and according to semver, the core release should probably be a major version - that means 11.0.0 - and as you say, that means every app would have to have a release to at least set max-version to 11 (and for some apps, min-version would also be 11.0.0). That is a significant downside to a major version number bump!

phil-davis · 2022-07-14T05:55:26Z

Can we add files_primary_s3 to the complete bundle? (does not directly help with the updater)
Can we push the complete bundle via online updater?

It's probably a good thing to add more of the supported apps to the complete bundle anyway - so that would help some cases in the future.

Maybe the online updater can be enhanced so that, when processing a core update, it also looks through the apps that are present in the installation (whether enable or disabled, doesn't much matter), and also updates all the apps to the latest version of each app that is compatible with the core version. Is there any reason that an algorithm to do that successfully can't be designed?

pmaier1 · 2022-07-14T08:09:57Z

Thanks, Phil. I understand that the right way to do it would be ownCloud 11. I'd really like to save the effort of releasing all apps now. Do we have other options?

phil-davis · 2022-07-14T08:37:15Z

Do we have other options?

make sure that all relevant apps have core min-version set to 10.11.0 (to ensure that older core installs will not suggesting updating to the new app version)
write good upgrade instructions for sites that do a "manual" upgrade. They need to put their system into maintenance mode, then update the core code, and update all their apps to the latest versions (specially the ones with Guzzzle7 releases), and only then exit maintenance mode.
investigate if https://github.com/owncloud/updater/ can be enhanced so that it will also look for and apply app updates in a single auto-update pass. (I guess that it's code is quite independent from the core code, so it should be able to "do what it likes" to the core+apps code folders without getting stuck in a Catch-22 situation) It might need some code smarts added so that it can do marketplace requests and downloads... like the market app, but independently.

jnweiger · 2022-07-16T18:25:39Z

More details for the update case:
Testing with marketplace.staging.owncloud.services, where files_primary_s3 2.0.0-rc.1 is available as version 2.0.0

The market-app in 10.10.0 hides files_primary_s3 2.0.0 and claims 1.2.0 is the latest available update. Probably OK.
config.php can have a setting
```
/**
 * Define whether or not to enable automatic update of market apps
 * Set to `false` to disable.
 */
'upgrade.automatic-app-update' => true,
```
that looks like it could make a difference during occ upgrade. It does not. files_primary_s3 remains at 1.2.0
Maybe it is meant to only update the market app? Wording of the comment is a bit unclear.
when the admin user was not logged out before the upgrade, he can log in after the upgrade.
but he cannot access any files:

In case the admin user can log in to the web-UI, he can see that there is now an update to files_primary_s3 2.0.0 available:

Clicking [UPDATE] fixes the situation, he can then access files again.
A complete update can be done from the command line like this:
```
occ upgrade
occ market:upgrade --all --major
```
Config.sample.php documents a setting occ config:system:set upgrade.automatic-app-update --type boolean --value true
It seems like it should make the manual market:upgrade unnecessary. It does not. BAD.

phil-davis · 2022-07-18T05:46:40Z

Config.sample.php documents a setting occ config:system:set upgrade.automatic-app-update --type boolean --value true
It seems like it should make the manual market:upgrade unnecessary. It does not. BAD.

It already defaults to true so there will be no different behavior if it is explicitly set to true in config.php

It was added here: https://github.com/owncloud/core/pull/28812/files

I suspect that there is a catch-22 for installations with files_primary_s3 which is enabled as the "real" storage. After putting the 10.11.0 core code in place, access to user's storage is broken, because the old files_primary_s3 app code is still there.

I wonder if we could make a version of files_primary_s3 code that detects which Guzzle is available in core, and makes the appropriate Guzzle calls. Then we could release that for 10.10 and people could update to that first. Then update core to 10.11.

phil-davis · 2022-07-18T10:26:30Z

I wonder if we could make a version of files_primary_s3 code that detects which Guzzle is available in core, and makes the appropriate Guzzle calls. Then we could release that for 10.10 and people could update to that first. Then update core to 10.11.

@jnweiger IMO that works - see PR #583

Can some devs review that please @janackermann @JammingBen @jvillafanez maybe? to see if I missed anything obvious.

phil-davis · 2022-07-18T10:49:26Z

@jnweiger please make a new release v2.0.0-rc.2 that has the latest files_primary_s3 master code after PR #583 was merged.
And try an upgrade path by:

install oC10.10 with the existing files_primary_s3
update files_prinary_s3 to the RC.2 release
check that oC10.10 still works
upgrade to oC10.11
check that "everything" still works

If it's possible/easy also try starting from oC10.9, make sure that the rc.2 works with core oC10.9. I suspect that there will be various sites that go from earlier oC10.9 10.8 ... and upgrade straight to 10.11

phil-davis · 2022-07-18T10:51:47Z

Actually, I think that we can call this next files_primary_s3 release 1.3.0 - it will work with oC10.10 (and before) and oC10.11 (and after). So maybe there is no need to bump the major version number.

jnweiger · 2022-07-18T22:38:28Z

https://github.com/owncloud/files_primary_s3/releases/tag/v2.0.0-rc.2 ready for some testing tomorrow morning.
It'd be awesome, if you achieved full compatibilitiy there! If so, yes, we'll release it as 1.3.0

phil-davis · 2022-07-22T07:26:47Z

@kiranadh1452 please test some upgrade paths, like:

Start with core oC10.9 and files_primary_s3 at the version 1.2.0 tag https://github.com/owncloud/files_primary_s3/releases/tag/v1.2.0
Have it set up and working
Check the latest master files_primary_s3 and composer install to get the dependencies up-to-date
Check that the S3 storage is still working
Update core to 10.10 and composer install to get the dependencies up-to-date, and php occ upgrade
Check that the S3 storage is still working
Update core to the current master (has version 10.11.0) and composer install to get the dependencies up-to-date, and php occ upgrade
Check that the S3 storage is still working

It will be fine for now to just use the cloned git repos, that will confirm that the code is working. Testing with tarballs can be done after release candidates are published.

We want to get confidence that the current files_primary_s3 master code works with multiple oC10 previous releases and the current master code. That will make the release and upgrade workflows much nicer.

kiranadh1452 · 2022-07-26T04:28:12Z

core: v10.9
files_primary_s3: v1.2.0

Primary S3 Storage Setup: ✔️
File Upload: ✔️
File Download: ✔️
File Deletion: ✔️
File/Folder Renaming: ✔️
Federated Sharing of Files and Folders in S3 Storage: ✔️

core: v10.9
files_primary_s3: latest(master)

Primary S3 Storage Setup: ✔️
File Upload: ✔️
File Download: ✔️
File Deletion: ✔️
File/Folder Renaming: ✔️
Federated Sharing of Files and Folders in S3 Storage: ✔️

core: v10.10
files_primary_s3: latest(master)

Primary S3 Storage Setup: ✔️
File Upload: ✔️
File Download: ✔️
File Deletion: ✔️
File/Folder Renaming: ✔️
Federated Sharing of Files and Folders in S3 Storage: ✔️

core: latest(master)
files_primary_s3: latest(master)

Primary S3 Storage Setup: ✔️
File Upload: ✔️
File Download: ✔️
File Deletion: ✔️
File/Folder Renaming: ✔️
Federated Sharing of Files and Folders in S3 Storage: ✔️

phil-davis · 2022-07-27T05:42:58Z

@jnweiger IMO this is solved by the current code in files_primary_s3 master. @kiranadh1452 has verified above.

Please try making a new RC tarball, and try it yourself. If it works, then we can move forward to make a new release of files_primary_s3 that should be published before we release core 10.11

jnweiger · 2022-07-27T23:14:39Z

I've tested files_primary_s3 1.3.0-rc.2 with core 10.8, 10.9.1, 10.10.0, and 10.11.0-alpha.2
files_primary_s3 1.3.0 installs fine via marketplace. on all these systems.

Updates tested

10.8 -> 10.11
10.9 -> 10.11
10.9 -> 10.10 -> 10.11
10.10 -> 10.11
In all update cases, primary s3 remained functional, and runs with guzzle MAJOR_VERSION=7 after the upgrade.

Odd observation:
files_primary_s3 2.0.0 (same content as 1.3.0, but min-version 10.11) was also always available on the staging marketplace before during and after upgrades. It never showed up.

occ upgrade --major did not do the trick, and clear cache of the market app did not do the trick.
Maybe something recognizes 10.11.0-alpha.2 to be less than the required 10.11 min-version?

phil-davis · 2022-07-28T03:17:45Z

https://www.php.net/manual/en/function.version-compare.php
"Note that pre-release versions, such as 5.3.0-dev, are considered lower than their final release counterparts (like 5.3.0). "

The problem/challenge/feature is that 10.11.0-alpha is considered to be before 10.11.0 (which it is actually, in the release process). I had not thought about that before. But core min-version was set to "10.11", and PHP version_compare is happy that 10.11.0-alpha.2 is greater than 10.11.

Maybe the marketplace code is automagically adding ".0" to "10.11" before doing version_compare - that would cause an issue. I will have a look...

Update: I don't easily see where this is going wrong, but I can't get the marketplace to return me any newer version of files_primary_s3. I locally changed my core to 10.11.1 and then:
occ market:upgrade --major files_primary_s3
And still it does not offer me any files_primary_s3 version 2.

Now I am not sure what is really on the marketplace, so maybe I can't debug this anyway.

jnweiger · 2022-08-03T07:32:34Z

Ah, sorry, the public marketplace does not have it. I did all the tests on marketplace.staging.owncloud.services instead.

phil-davis mentioned this issue Jul 18, 2022

Allow Guzzle5 or Guzzle7 to exist in core #583

Merged

phil-davis mentioned this issue Jul 27, 2022

Release 1.3.0 for Guzzle7 #577

Closed

42 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QA] Update to 10.11.0 kills primary S3 storage #582

[QA] Update to 10.11.0 kills primary S3 storage #582

jnweiger commented Jul 13, 2022 •

edited

Loading

jnweiger commented Jul 13, 2022 •

edited

Loading

phil-davis commented Jul 14, 2022 •

edited

Loading

phil-davis commented Jul 14, 2022

pmaier1 commented Jul 14, 2022

phil-davis commented Jul 14, 2022

jnweiger commented Jul 16, 2022 •

edited

Loading

phil-davis commented Jul 18, 2022

phil-davis commented Jul 18, 2022

phil-davis commented Jul 18, 2022 •

edited

Loading

phil-davis commented Jul 18, 2022

jnweiger commented Jul 18, 2022

phil-davis commented Jul 22, 2022

kiranadh1452 commented Jul 26, 2022

phil-davis commented Jul 27, 2022

jnweiger commented Jul 27, 2022 •

edited

Loading

phil-davis commented Jul 28, 2022 •

edited

Loading

jnweiger commented Aug 3, 2022

[QA] Update to 10.11.0 kills primary S3 storage #582

[QA] Update to 10.11.0 kills primary S3 storage #582

Comments

jnweiger commented Jul 13, 2022 • edited Loading

jnweiger commented Jul 13, 2022 • edited Loading

phil-davis commented Jul 14, 2022 • edited Loading

phil-davis commented Jul 14, 2022

pmaier1 commented Jul 14, 2022

phil-davis commented Jul 14, 2022

jnweiger commented Jul 16, 2022 • edited Loading

phil-davis commented Jul 18, 2022

phil-davis commented Jul 18, 2022

phil-davis commented Jul 18, 2022 • edited Loading

phil-davis commented Jul 18, 2022

jnweiger commented Jul 18, 2022

phil-davis commented Jul 22, 2022

kiranadh1452 commented Jul 26, 2022

phil-davis commented Jul 27, 2022

jnweiger commented Jul 27, 2022 • edited Loading

phil-davis commented Jul 28, 2022 • edited Loading

jnweiger commented Aug 3, 2022

jnweiger commented Jul 13, 2022 •

edited

Loading

jnweiger commented Jul 13, 2022 •

edited

Loading

phil-davis commented Jul 14, 2022 •

edited

Loading

jnweiger commented Jul 16, 2022 •

edited

Loading

phil-davis commented Jul 18, 2022 •

edited

Loading

jnweiger commented Jul 27, 2022 •

edited

Loading

phil-davis commented Jul 28, 2022 •

edited

Loading