Provide API endpoints for favorites/metadata/tags on files

[PVince81]

File list

Goal is to make it possible to receive a list of favorite files and/or ids to be able to display them.
Ideal case would be to have everything go over a OCS API and use it internally as well.

Here are three alternatives about how to retrieve the list of files

1. Extend list.php

Currently the list.php ajax call already returns a list of files.
That list could be made filterable by tag using "?tag=tag1" and also make it return a "tags" attribute as part of the result set:

[{
    "id": "8",
    "name": "test",
    "path": "/path/to/subdir",
    "parentId": "3",
    "date": "November 19, 2014 at 11:16:30 AM GMT+0",
    "etag": "546c7c0ed4527",
    "icon": "/owncloud/core/img/filetypes/folder.svg",
    "mimetype": "httpd/unix-directory",
    "mtime": 1416395790000,
    "permissions": 31,
    "size": "123",
    "tags": ["tag1", "_$!<Favorite>!$_"]
}]

This approach would need a db JOIN with the "vcategory" tables when requesting the files list, might be possible.

The tags would be included in the WebDAV response as extended attributes.

Advantage: all the information is already there, reusing existing endpoint (WebDAV + internal)
Disadantage: performance impact is unclear

2. Provide new OCS API endpoints to retrieve the information

• a) Get the list of files by tag: GET /ocs/v1.php/apps/files/api/v1/tags/{tagname}?path=path/to/subdir&recursive=false
• b) Get the list of favorite files: GET /ocs/v1.php/apps/files/api/v1/favoritefiles?path=path/to/subdir&recursive=false
• c) Get the list of favorite file ids: GET /ocs/v1.php/apps/files/api/v1/favoritefiles?idonly=true&path=path/to/subdir?recursive=false

After rendering the file list with "ajax/list.php" the code must also call the "favoritefiles?idonly=true&path=currentpath" endpoint to get the id of all files inside that folder that have the favorite/tag (similar to what OC.Share does at the moment).

Advantages: No JOIN needed, tags API separate from files (if that's an advantage)
Disadvantages: Cross-reference of files+tags need to be done on the JS side (and API consumer). An external API consumer will need to call WebDAV first to get the file ids, then map it with the result of the tags call.

Tags list

Additional APIs are required to manage the tags themselves:
a) Get all tags for the current user: GET /ocs/v1.php/apps/files/api/v1/tags?category=files
b) Delete a specific tag for the current user: DELETE /ocs/v1.php/apps/files/api/v1/tags/{tagname}

Assigning tags to files

Two possibilities:

1. REST style with PATCH:

If we want to do it REST style, we should use the PATCH method and patch the files metadata:
PATCH /ocs/v1.php/apps/files/api/v1/files/path/to/file and pass a JSON object {tags: ['newtag1', 'newtag2']}

Advantage: REST style
Disadvantage: the endpoint itself doesn't exist (addressing files with files) so it would be added solely for this purpose and also would not be possible to change any other file attribute using that method as it would be expected from a PATCH call (so, might be overkill). If we had a full REST API for the files list it might make sense. But we have WebDAV for that.

2. Non-REST:

Same as before but with "POST"
POST /ocs/v1.php/apps/files/api/v1/files/path/to/file and pass a JSON object {tags: ['newtag']}
or even less REST:
POST /ocs/v1.php/apps/files/api/v1/settags?path={filePath}&tags=newtag1,newtag2

3. "Reverse" REST

• Add tag on a given file: POST /ocs/v1.php/apps/files/api/v1/tags/{tagname}?path=file
• Delete tag from a given file: DELETE /ocs/v1.php/apps/files/api/v1/tags/{tagname}?path=file
• Delete all tags from a given file: DELETE /ocs/v1.php/apps/files/api/v1/tags?path=file

4. Webdav attributes

I heard once that it might be possible to set custom attributes over WebDAV and they would land in the oc_properties table. Maybe the same API/endpoint could be used for setting tags on files.

Advantages: integrated with WebDAV, no extra strange endpoing as for 1) and 2)
Disadvantages: the files app doesn't use WebDAV so it would need an additional internal endpoint that does exactly the same

[PVince81]

Also the question whether it makes sense to limit the "tags" API to just files for the files app that needs it, or make a more generic tags endpoint. For a generic one the issue would be that the result set could be a mix between files and other results (contacts, etc) for a single tag.

Maybe the tag management API itself should be generic (get all known tags, delete tags) but not the file-specific ones.

[PVince81]

Summonning @DeepDiver1975 @schiesbn @icewind1991 @ockham for advice/discussion

[PVince81]

Further ideas:

• Get all tags set on the given files: GET /ocs/v1.php/apps/files/api/v1/tags?path=path/to/subdir&recursive=false
• Get all file ids that have the given tag: GET /ocs/v1.php/apps/files/api/v1/tags/{tagname}?path=path/to/subdir&recursive=false

Next step to investigate:

1. How to do the JOIN / performance implications
2. How to get/set WebDAV properties

[PVince81]

CC @nickvergessen

[DeepDiver1975]

2. How to get/set WebDAV properties

this is how we do this as of today for other webdav props:
https://github.com/owncloud/core/blob/master/lib/private/connector/sabre/filesplugin.php#L77

[PVince81]

@DeepDiver1975 thanks, yes. I think we can return the tags as a new tag <o:tags>tag1,tag2</o:tags> (might need some encoding though)

On question I'll investigate is whether there's a way to use WebDAV to modify such properties, for example using a PATCH method on the file or use whichever method one can use to store stuff into "oc_properties".

[DeepDiver1975]

On question I'll investigate is whether there's a way to use WebDAV to modify such properties,

WebDAV used PROPSET and PROPGET verbs for this

or use whichever method one can use to store stuff into "oc_properties".

I would not go for that table - highly unstructured and will result in major performance penalties

[PVince81]

I thought about something like "if PROPSET is called with the name 'tags' then use the tags API instead, else store to oc_properties". But maybe that's too hacky ?

[PVince81]

Just to clarify: there is already a Tags API and vcategory tables for tags which are currently used by the Contacts app. We'll reuse that to store the files tags as well.

[DeepDiver1975]

Well - it's an overall design decision we have to take (now?):

Do we want the files app to become more and more a webdav client or not?

Pros:

• major code reduction on PHP code side
• possibility to use chunked upload in the browser as well
• as soon as we provide new functionalities to the files app - they are available to the clients (mobile/desktop) as well

Cons:

• js webdav client????
• ???

[PVince81]

Thing is, I'd love to make the files app use WebDAV only in the long run... we could write a JS WebDAV library that can be reused outside as well and use it in the files app... One drawback mentionned by @icewind1991 is that IE8 might not support WebDAV verbs like PROPFIND properly...

Also I believe that list.php is returning information that is currently not available in WebDAV, like "isPreviewAvailable" and others. So might be valid in the long run but not doable right now.

For now I'd extend "list.php" to be able to return an extra tags: ['tag1', 'tag2'] attribute and add a <oc:tags>tag1,tag2</oc:tags> in the WebDAV response, so no need for extra OCS API calls for getting the regular file list. We'll still need OCS calls to return the file list filtered by tags as WebDAV has no concept of filter (ex: return all files from all folders tagged with "tag1"), this is needed for the future "Favorites" file view accessible from the sidebar.

[PVince81]

@craigpg @karlitschek I assigned this task to the OC 8 milestone.

[PVince81]

Side note: at some point someone on the dev mailing list asked if there was a JS client for OC so I made this experiment: https://gist.github.com/PVince81/cdfb1287c7f2a2db5f25

[DeepDiver1975]

google gives some results for js webdav - e.g. https://github.com/sara-nl/js-webdav-client

[PVince81]

Could we use PROPPATCH to set the tags properties ? Or is that the same facility like PROPGET/PROPSET ?

[DeepDiver1975]

Could we use PROPPATCH to set the tags properties ? Or is that the same facility like PROPGET/PROPSET ?

afaik yes - not too sure to be honest - @dragotin should know ..

[PVince81]

I raised #12353 to research/discuss moving the files app to WebDAV in the future.

[PVince81]

I'm still kind of stuck here, what approach should I use ?

1. If we agree with having different WebDAV endpoints, I could use this: https://domain.tld/owncloud/remote.php/dav/tags/favoritetag/ and the "tags" endpoint with REST-style to manage tags. (WebDAV endpoint extension approach here: Replace WebDAV with REST #12504).

2. If we want OCS instead, then I'll need OCS adapter Make OCS routes work together with the AppFramework #12454 to be implemented so I can use the app framework. Else it means I'll need to ditch the app framework or use a hacky approach...

3. And as discussed so far, we don't want a non-OCS non-WebDAV REST API, so I'll abandon this branch of alternatives. (this is what I have in the current PR, need to change it)

4. Yet another idea would be to use WebDAV with the SEARCH method.
   Setting tags on files would still need to go with a POST or something.
   But tag management still doesn't have an endpoint in that case, so it would still go to OCS...

[DeepDiver1975]

while I really LOVE to have an REST api - I'd vote against option 3 - we need to manage the zoo of apis and not add more and more - http://www.commitstrip.com/en/page/3/

regarding option 4 - webdav search is a standard which afaik never got implemented anywhere - we had that some month back already in a customer project

we either go by option 1 or 2

option 2 contains work which anyhow needs to be done sooner or later just for the sake of code quality/testability - not much lost from a coding point of view

option 1 is very promising from an melting-technologies-together-pov - but requires some thinking on how to design this new webdav endpoint

I guess short term (OC8) will be option 2 and we continue to think about the pros/cons regarding option 1 these days/weeks

[PVince81]

Thanks a lot for your feedback, makes sense.

I'll move the PR to use OCS endpoints instead as soon as the upfront work is ready.
Still busy tweaking the JS and other parts this week.

[karlitschek]

Thanks you

[PVince81]

Let's keep this ticket here for the OCS endpoints (the current ones in #12360 are still REST, will change after merge)

I raised a separate ticket for returning the tags/favorite info as part of WebDAV properties, to be discussed here: #12856

[DeepDiver1975]

@PVince81 I'd like to close this ticket and continue with new ticket(s) from the remaining parts which will go into the direction of OC8.1/OC8.2.

What's missing here now?

[PVince81]

The favorites/tags of files can be retrieved with PROPFIND and set with PROPPATCH, so I'd say this task is completed.
At some point we'll want to have an API to return the list of all known tags, but that's another feature (and as long as we don't have any UI that lists tags, it might not be needed)

[melalitesh]

@PVince81 We tried setting Tag using PROPPATCH

Code: https://bpaste.net/show/6bd75b2d3512

But, on retrieving with PROPFIND we didn't get any tag
Response: https://bpaste.net/show/4cc7d9e7c00a

Please suggest,
How to set the tags using webdav API and retrieve them.

[PVince81]

@lalitesh-infoobjects these are not tags but DAV properties. This is not the same thing.

The tags here are about simple tags like "Favorite", "Private", "Important" labels.

The thing in your code is a custom DAV property, but your PROPPATCH syntax looks fine.
Check the oc_properties table in the database to see if the DAV property was set.

Also please try again with 8.1, I tested it there a while ago and it should work. Not sure about 8.0.x.

[melalitesh]

@PVince81 : Updated to 8.1 rc1
Updated the sabre / guzzle clients
Checked the oc_properties table, we can see the properties created their. But DAV properties are not included in PROPFIND response.

Response:
https://bpaste.net/show/4a9479bb99a1

[PVince81]

Please create a new ticket for this issue. Thanks.
Note that DAV properties need to be requested explicitly in the PROPFIND request. They do not appear by default.