Allowing access to Swift containers shared by ACL

[MorrisJobke]

To avoid a confusing extra field, the bucket field is overloaded to
optionally take a URL.

This is just a copy of #19003 to have it run at our CI.

@PVince81 already 👍ed this: #19003 (comment)

@butonic @icewind1991 @Xenopathic Want to review this?

[mention-bot]

By analyzing the blame information on this pull request, we identified @berendt, @PVince81 and @icewind1991 to be potential reviewers

[MorrisJobke]

@tjdett Can we get a license statement on your patch for this? We either need MIT license or a signed CLA - https://owncloud.org/contribute/agreement/

Thanks for your patch and sorry that this was on hold for so long.

[tjdett]

@MorrisJobke I have personal authority to licence this as MIT, but the contributor agreement for myself and @DanielTosello is still stuck in our legal department.

If MIT, how would you like me to indicate it? In the code or the commit?

[LukasReschke]

If MIT, how would you like me to indicate it? In the code or the commit?

Please leave a comment stating that this contribution is MIT licensed.

[tjdett]

The commits in the following pull-requests are provided under the MIT Licence:

• Don't log expected Swift 404 responses in files_external #19000
• Improving fopen behaviour for Swift in files_external #19001
• Streaming download from Swift in files_external #19002
• Allowing access to Swift containers shared by ACL in files_external #19003 and the rebased version Allowing access to Swift containers shared by ACL #21008
• Handle files which require Swift large object storage in files_external #19004

Copyright (c) 2015 The University Of Melbourne

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sellcovered
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

This also applies to any alterations to the listed pull-requests provided by @DanielTosello on behalf of the University of Melbourne.

[LukasReschke]

@DeepDiver1975 core-ci-linux-swift-primary-storage/database=mysql,label=SLAVE somehow seems to make problems, care to check? THX

11:00:30 Started by upstream project "core-ci-linux-swift-primary-storage" build number 40118
11:00:30 originally caused by:
11:00:30  Started by an SCM change
11:00:30 Building remotely on lex-j1 (SLAVE) in workspace /ssd/jenkins/workspace/core-ci-linux-swift-primary-storage/database/mysql/label/SLAVE
11:00:31  > git rev-parse --is-inside-work-tree # timeout=10
11:00:31 Fetching changes from the remote Git repository
11:00:31  > git config remote.origin.url https://github.com/owncloud/core.git # timeout=10
11:00:31 Fetching upstream changes from https://github.com/owncloud/core.git
11:00:31  > git --version # timeout=10
11:00:31 using GIT_SSH to set credentials 
11:00:31 using .gitcredentials to set credentials
11:00:31  > git config --local credential.username jenkins # timeout=10
11:00:31  > git config --local credential.helper store --file=/tmp/git6317423094818872118.credentials # timeout=10
11:00:31  > git -c core.askpass=true fetch --tags --progress https://github.com/owncloud/core.git +refs/heads/*:refs/remotes/origin/*
11:00:57  > git config --local --remove-section credential # timeout=10
11:00:57 Checking out Revision 4fe32bed45aef87874e969d577f14a238f12c5e5 (origin/UoM-ResPlat-DevOps-swift-acl-shared-containers)
11:00:57  > git config core.sparsecheckout # timeout=10
11:00:57  > git checkout -f 4fe32bed45aef87874e969d577f14a238f12c5e5
11:00:58 FATAL: Could not checkout 4fe32bed45aef87874e969d577f14a238f12c5e5
11:00:58 hudson.plugins.git.GitException: Could not checkout 4fe32bed45aef87874e969d577f14a238f12c5e5
11:00:58    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1907)
11:00:58    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
11:00:58    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
11:00:58    at hudson.remoting.UserRequest.perform(UserRequest.java:120)
11:00:58    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
11:00:58    at hudson.remoting.Request$2.run(Request.java:326)
11:00:58    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
11:00:58    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
11:00:58    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
11:00:58    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
11:00:58    at hudson.remoting.Engine$1$1.run(Engine.java:62)
11:00:58    at java.lang.Thread.run(Thread.java:745)
11:00:58    at ......remote call to lex-j1(Native Method)
11:00:58    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
11:00:58    at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
11:00:58    at hudson.remoting.Channel.call(Channel.java:781)
11:00:58    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
11:00:58    at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source)
11:00:58    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:00:58    at java.lang.reflect.Method.invoke(Method.java:606)
11:00:58    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
11:00:58    at com.sun.proxy.$Proxy69.execute(Unknown Source)
11:00:58    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1060)
11:00:58    at hudson.scm.SCM.checkout(SCM.java:485)
11:00:58    at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
11:00:58    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
11:00:58    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:00:58    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
11:00:58    at hudson.model.Run.execute(Run.java:1738)
11:00:58    at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
11:00:58    at hudson.model.ResourceController.execute(ResourceController.java:98)
11:00:58    at hudson.model.Executor.run(Executor.java:410)
11:00:58 Caused by: hudson.plugins.git.GitException: Command "git checkout -f 4fe32bed45aef87874e969d577f14a238f12c5e5" returned status code 128:
11:00:58 stdout: 
11:00:58 stderr: fatal: cannot rmdir 'tests/objectstore/entrypoint.sh': Permission denied
11:00:58 
11:00:58    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640)
11:00:58    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:62)
11:00:58    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1899)
11:00:58    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
11:00:58    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
11:00:58    at hudson.remoting.UserRequest.perform(UserRequest.java:120)
11:00:58    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
11:00:58    at hudson.remoting.Request$2.run(Request.java:326)
11:00:58    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
11:00:58    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
11:00:58    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
11:00:58    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
11:00:58    at hudson.remoting.Engine$1$1.run(Engine.java:62)
11:00:58    at java.lang.Thread.run(Thread.java:745)
11:00:58 Recording test results
11:00:58 ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did tests run? 
11:00:58 For example, /ssd/jenkins/workspace/core-ci-linux-swift-primary-storage/database/mysql/label/SLAVE/tests/autotest-results-mysql.xml is 3 days 2 hr old
11:00:58 
11:00:58 Setting commit status on GitHub for https://github.com/owncloud/core/commit/4fe32bed45aef87874e969d577f14a238f12c5e5
11:00:59 Finished: FAILURE

[MorrisJobke]

@DeepDiver1975 @LukasReschke @Xenopathic @butonic I guess there is a difference between the primary storage object store and the files_external object store, right? @Xenopathic Is there already an objectstore files_external test setup?

[PVince81]

@MorrisJobke this PR only affects files_external for external storage mount points.
The code for the home object store is here: https://github.com/owncloud/core/blob/master/lib/private/files/objectstore/swift.php

[PVince81]

I merged this with master and ran the unit tests locally, they passed:

PHPUnit 4.3.5 by Sebastian Bergmann.

Configuration read from /srv/www/htdocs/owncloud/tests/phpunit-autotest-external.xml

S................................................................ 65 / 78 ( 83%)
.............

Time: 35.95 seconds, Memory: 25.25Mb

There was 1 skipped test:

1) Test\Files\Storage\Swift::testStat
Swift doesn't update the parents folder mtime

/srv/www/htdocs/owncloud/apps/files_external/tests/backends/swift.php:71

OK, but incomplete, skipped, or risky tests!
Tests: 78, Assertions: 406, Skipped: 1.

Let me rebase this branch to let CI have another go.

[PVince81]

Rebased again, CI was stuck 😦

@icewind1991 @Xenopathic please review the code change

[icewind1991]

Changes look good 👍