Introduce CI for files_external

[PVince81]

@DeepDiver1975 @karlitschek @LukasReschke @icewind1991
@craigpg @MTRichards

Whether we keep files_external in core or not it doesn't matter.
Most importantly we need to setup a CI solution for files_external (external storage app).

Let's discuss it here.
Follow up from #8002

[LukasReschke]

I couldn't agree more - especially considering that just by luck @PVince81 noticed today that our SFTP unit tests are simply failing: #11692

[PVince81]

And also a lot of the S3 tests fail as well.

[karlitschek]

ok. let´s do that. but it has to stay in the core repos because of other reasons.

[DeepDiver1975]

I'll put this on my agenda then

[PVince81]

We could use docker for some of the storage instances (FTP, SFTP, SMB, etc)
That docker part could even be included in "autotest.sh". If docker is detected, then pull some images, start them and run the tests locally.

[PVince81]

If we agree with the docker approach, I'll try to play around with it a bit, for FTP as a proof of concept.
Having these basic storage types covered would already be a small step forward.

For other storages like S3 and SWIFT we'll probably need an already running server (there's no docker image for these yet, and devstack is HUGE)

Please note that docker instances can be reset/recycled without having to re-download the whole image every time. This, if the Jenkins images are not completely reset between runs.

[DeepDiver1975]

This, if the Jenkins images are not completely reset between runs.

our jenkins deplyoment has no virtualization of any kind in place - nothing will be reset

[DeepDiver1975]

That docker part could even be included in "autotest.sh".

This docker based approach sounds reasonable - but requires analysis to determine if we can fire up docker on the current infrastructure.

But this is the way to go from my current understanding.

[DeepDiver1975]

That docker part could even be included in "autotest.sh"

And I'd really go for a new script - autotest-external.sh where we can specify the storage type as argument.

[PVince81]

Yes, that sounds good.
The script could be put inside of apps/files_external/tests.
There's already a "config.php" there with disabled configs, these could be changed to point to the assumed localhost docker instances.
By default the script would run all enabled tests.
If one backend name is specified, it will limit the test to that one.

At least the first step would be to make these runnable locally without requiring the devs to manually setup extra envs (apart from docker itself).

I'll leave the infrastructure question for you 😉
We can always provide a setting somewhere to tell it to not run against local docker instances but use remote ones (an alternative config file)

[PVince81]

So here's the possible plan:

1. Provide/find docker images (when possible) for every external storage instance
2. Write shell scripts for each storage: to pull/run docker image, configures/run the external storage tests, then cleans up
3. Write a master script autotest-external.sh that runs the scripts from 2) (with possibility to disable tests)
4. Once most tests can be run locally with a single script run, look into setting them up on a CI server
5. Then look into the remaining non-docker ext storages, these might need direct config on the CI server / ecosystem

Also:

• if docker is not found, ask for it to be installed or point to manual install
• manual (non-docker) config must still be possible as it was in the past

CC @MorrisJobke I heard you like docker ? 😉

[PVince81]

I'd say we need at least the master script and an example docker instance, then we can get help to setup the other instances / make separate tickets for them.

[MorrisJobke]

I will check for possible ways to bring docker instances of files_external backends to a CI ready state. I will check that tomorrow.

[PVince81]

Let's start with one, then need to think of a structure for the scripts.
I could do that if you like, so you can focus on docker.

[DeepDiver1975]

Can docker run inside another virtual machine? This is critical as we do not have a physical server (yet) in case needed.

[MorrisJobke]

@DeepDiver1975 Yes. Docker is just a combination between set of available filesystem (and therefore executables and stuff that is in a filesystem) with a defined process which runs inside of this filesystem. And it is really lightweight, because it's just an encapsulated process running on the hosts kernel.

[DeepDiver1975]

thx @MorrisJobke - what will be our first external storage we want to test against? webdav?

[MorrisJobke]

@DeepDiver1975 I just created a FTP container locally

[MorrisJobke]

FTP: #14076

[PVince81]

• Regular WebDAV raised as: WebDAV docker instance for autotest-external #15897

[ghost]

@DeepDiver1975 ping. For 8.2 or later?

[RobinMcCorkell]

To be done:

• Amazon S3 (I hope to make a test tomorrow)
• Dropbox
• Google
• More FTP servers (they all behave differently)
• Alternative WebDAV server(s) WebDAV docker instance for autotest-external #15897
• SFTP public key
• Swift [files_external] swift tests #14077

[DeepDiver1975]

9.0

[PVince81]

Nice, we did make some progress. I just ticked swift 😄

[PVince81]

SFTP is close to working with two failing tests (PRs included): #11692

Webdav was merged a while ago, could be put into CI as well: #16688

[PVince81]

@DeepDiver1975 close this ? We want to move everything to apps anyway so CI should be added there when doing just that.

[PVince81]

Closing this. We should focus on moving ext storage backend individually to separate apps and add CI there when possible.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.