How do we migrate the namespace?

[butonic]

OC10 vs CS3 namespaces

In oc10 there are two webdav endpoints:

• remote.php/webdav: the 'old' endpoint that presents the logged in users home directory
• remote.php/dav: the 'new' endpoint that presents several sub namespaces
  • remote.php/dav/avatars: for avatars
  • remote.php/dav/files/<username>: renders that users home directory
  • remote.php/dav/meta/<fileid>/v: to access versions
  • remote.php/dav/trash-bin: for trash

The CS3 namespace, similar to our 'new' remote.php/dav endpoint does not immediately render the users home directory, but allows the admin to provision and mount multiple storages. In the CS3 namespace the users home might be located at /eos/cern.ch/users/hugo. For convenience a 'virtual' home storage provider is mounted at /home which will render the same files. Projects might be mounted at /projects and incoming and outgoing shares under /myshares. Admins can mount different storage zones and expose them using /zones/a, /zones/b ... whatever.

This is by design and allows us to get rid maintaining metadata in a central database.

The problem

This creates an interesting problem when migrating from oc10 to ocis: we need to decide what namespace to present at the two ownCloud webdav endpoints.

If we continue to jail users into their home, as we do today, we need to add different namespaces like /projects or /spaces by hardcoding new endpoints into the clients to make them usable.

If we create a new namespace that transparently presents the cs3 namespace existing clients can access the full namespace. Unfortunately, it causes the paths of files to change. AFAICT they will discover that the files have all been moved to a subfolder, which will recreate the new path prefix to their home in the CS3 namespace on their local disk. Which I assume is not what they expect. There be dragons...

Some more background

Currently, the desktop client uses the new webdav endpoint only if the dav.chunking capability is >1.0. Which is the case for all oc10 installations ... but the one at CERN, which does not support chunkingng, which is why clients are using the old webdav endpoint. CERN also has replaced the user tree there with a cs3 like namespace a long time ago, that is how their end users can get access project storages or other users files with sharing implemented via ACLs.

All other clients currently use the old webdav endpoint to list files.

Possible solutions

0. CERN has no problem, they just present the CS3 namespace at the new and the old namespace. It is what their clients already know. They are already set up with selective syncs for the users home and, if setup, other project folders.

1. Existing installations can be oblivious of the change and we can just jail them in their home at both endpoints. If they want to access other endpoints they need to use the cs3 api directly.

2. We can expose the CS3 namespace at the old webdav endpoint only. This would allow browsing the complete CS3 namespace in the browser and mobile clients, while the desktop clients will continue to sync against the new webdav endpoint, still being jailed into the users home.

3. We can expose the CS3 namespace at both endpoints, causing the desktop client to move files into a sub directory unless we teach it a migration trick.

4. We introduce a new /dav/cs3 endpoint that presents the cs3 namespace. The existing endpoints are both still jailed to the users home (or at CERN not). We teach all clients how to use the new endpoint and allow them to implement a better migration.

I think 4. is the best way forward. For development we can present the cs3 namespace at the new and old webdav endpoint in reva/ocis. Adding a /dav/cs3 endpoint to reva is trivial. The migration is more challenging.

Feedback welcome.

...

[davigonz]

In the case of Android:

all mobile clients present a virtual filesystem and should be able to deal with this gracefully

Hmm, yes, a good example of this is when we configure an External storage (Google Drive, SFTP...) for our ownCloud account. We show that external storage as one more folder in the Android app, along with the rest of files and folders.

However I am not sure what happens with files that are kept in sync. I admit, I don't even know if they are tracked by file id or by path

Let's say that the synchronization for kept in sync files consists of:

1. We get the local file from database using the remote path.
2. We create a remote file object after getting it from the server.
3. We compare the timestamps and etags of those two objects and decide if the file changed in the server o changed locally.
4. We perform the corresponding sync operations (upload, download..) depending on where the latest changes in the files took place, as long as there's no conflict between the local and server files.

[butonic]

Another question I have for sync developers:

The etag will no longer be propagated to the root of the CS3 namespace. It contains an instance wide namespace so the root would reflect changes of all users. Etags will only be propagated to the root of the users home storage or more generally, to the top of a subtree.

Will this work with selective sync? So, can the desktop client detect changes if only a sub tree is synced?

[butonic]

Related: owncloud/owncloud-sdk#349 (comment)

[butonic]

@guruz @ogoffart what happens when a selective sync folder root is moved to a different parent on the server side? will the client do a discovery at the root and detect the new mount point?

[ogoffart]

@butonic I'm not sure you are using the terminology "selective sync" the way it is in the client. The selective sync in the client is the fact that the user can choose what to sync by un-selecting some folder so that these are never synchronized. This used path name and as long as the name of the path does not change, it will be kept unsynchronized. etag don't matter. If the folder is moved to a new path this will break the user selection. (the new path will now be synchronized)

However, for the normal sync, the etag of a directory need to change or we do not look into that directory for changes. So if the etag of remote.php/dav/files// does not change, the sync will not happen.

[butonic]

@ogoffart hm then I am misusing selective sync. I mean being able to set up multiple sync folders, eg: sync http://localhost:20080/remote.php/dav/files/jfd/Projects to my C:/Users/jfd/Projects folder and http://localhost:20080/remote.php/dav/files/jfd/Documentation to my D:/Docs folder. What is that feature called? And what happens if http://localhost:20080/remote.php/dav/files/jfd/Documentation is moved to http://localhost:20080/remote.php/dav/files/jfd/Archive/2019/Documentation? Will that break the sync? Or does the client try to detect a rename?

[labkode]

@butonic to make migration easier, the endpoint we currently use at remote.php/webdav we can skip it, maybe it will also simplify code in the ocdav service.

We are still using it but we want to move forward using the new protocol used by the oc sync client and this legacy one. So the migration should focus only on the new namespace.

remote.php/dav: the 'new' endpoint that presents several sub namespaces

• remote.php/dav/avatars: for avatars
• remote.php/dav/files/<username>: renders that users home directory
• remote.php/dav/meta/<fileid>/v: to access versions
• remote.php/dav/trash-bin: for trash

These are the ones that should be considered, I hope this will make migration simpler.

• remote.php/dav/files/hugo can easily be mapped to a home storage provider in Reva.
• remote.php/dav/trash-bin to the trashbin in the home storage provider, we'll discuss what should be do when multiple storage providers are available for a user, let's start simple.
• remote.php/dav/meta/<fileid>/v: to the versions of the home storage provider. As the trash-bin we start simple.
• remote.php/dav/avatars to a storage provider or some other metadata CS3 API, not clue yet.

What do you think?

[butonic]

@ogoffart does the client allow using the vfs with the old webdav endpoint remote.php/webdav/?

[ogoffart]

And what happens if http://localhost:20080/remote.php/dav/files/jfd/Documentation is moved to http://localhost:20080/remote.php/dav/files/jfd/Archive/2019/Documentation? Will that break the sync? Or does the client try to detect a rename?

That will break the sync. Wou will have to re-create your sync folder.

does the client allow using the vfs with the old webdav endpoint remote.php/webdav/?

Yes. I think so.

[madsi1m]

@butonic i like solution 4 the most. as to the user most things will be in the same place. Or modify /dav to present all the storages a user can see and black list some used names like "home", "trash-bin" etc

[butonic]

Clients will need to discover if a global namespace is available and where to find it. adding yet more urls to query is counterproductive: cs3org/cs3apis#33

@DeepDiver1975 Maybe the graph api can be used to simplify the lookup?

[labkode]

@butonic @madsi1m I also like the point 4 the most however it requires changes in the clients.

Adding support for the "dav/cs3" endpoint propossed by @butonic sounds really good and it is in the spirit how ownCloud has been handling different namespaces (remote.php/webdav/ and remote.php/dav/files) by using capabilities.

So if we add a new capability to enable the cs3 support then the clients will be configured to talk to it.

So if we go for point 4 the questions we need to ask ourselves is what minimum layout will be needed:

cs3/home for users homes
cs3/global to autodiscover all available storage providers and we can expose all of them (similar to the storage-root but in a virtual approach)
cs3/global/users/ to store all users 
cs3/global/projects/ store all projects
cs3/global/mystuff not particularly meaning but exposed by organization deploying OCIS.

Having such name-spacing will help during migration as:

• remote.php/webdav/ can be mapped to /cs3/home or /cs3/global (to consider the implications)
• remote.php/dav/files/doe can be mapped to /cs3/global/users/doe

The rest of the dav endpoints (trashbin and meta) lives only on the ocdav layer as there is no reason to expose this information as a storage. Relying on the cs3apis to query trashbin or versions suffice.

We can iterate over this face to face next two weeks, what do you think?

[madsi1m]

Sounds great.

When a user shares a folder to another user, is it intended to show up in home, projects, “mystuff”, or somewhere else?

[dragotin]

Please see owncloud/ocis#1827 for retrieving available spaces for a user.

[michaelstingl]

Related:

• [Spaces] [Desktop] OCIS Spaces Migration API ocis#3574