Expose configuration for the reva archiver #2509

jvillafanez · 2021-09-15T14:44:01Z

Description

Expose the reva archiver configuration

Related Issue

Motivation and Context

How Has This Been Tested?

Access ocis with the admin user
Create folder "zzz" and upload some files there
Connect to the archiver service:
1. Assuming you've started a basic ocis installation with ocis server
2. In the same host

root@f13b05e8714d:/# curl -v -u admin:admin -JO http://localhost:9140/archiver?dir=/home/zzz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:9140...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9140 (#0)
* Server auth using Basic with user 'admin'
> GET /archiver?dir=/home/zzz HTTP/1.1
> Host: localhost:9140
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.68.0
> Accept: */*
> 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Disposition: attachment; filename="zzz.tar"
< Content-Transfer-Encoding: binary
< Vary: Origin
< X-Access-Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjMxNzk4NDYxLCJpYXQiOjE2MzE3MTIwNjEsImlzcyI6Imh0dHBzOi8vb2Npcy5qcC5zb2xpZGdlYXIucHJ2IiwidXNlciI6eyJpZCI6eyJpZHAiOiJodHRwczovL29jaXMuanAuc29saWRnZWFyLnBydiIsIm9wYXF1ZV9pZCI6ImRkYzIwMDRjLTA5NzctMTFlYi05ZDNmLWE3OTM4ODhjZDBmOCIsInR5cGUiOjF9LCJ1c2VybmFtZSI6ImFkbWluIiwibWFpbCI6ImFkbWluQGV4YW1wbGUub3JnIiwiZGlzcGxheV9uYW1lIjoiQWRtaW4iLCJ1aWRfbnVtYmVyIjoyMDAwNCwiZ2lkX251bWJlciI6MzAwMDB9LCJzY29wZSI6eyJ1c2VyIjp7InJlc291cmNlIjp7ImRlY29kZXIiOiJqc29uIiwidmFsdWUiOiJleUp3WVhSb0lqb2lMeUo5In0sInJvbGUiOjF9fX0.jf3Fxy5FGRLLKLss_gi8yxZ1MVof2-S3g17z4xtL_KA
< Date: Wed, 15 Sep 2021 13:21:02 GMT
< Content-Type: application/octet-stream
< Transfer-Encoding: chunked
< 
{ [3248 bytes data]
100 6782k    0 6782k    0     0  9566k      0 --:--:-- --:--:-- --:--:-- 9552k
* Connection #0 to host localhost left intact
curl: Saved to filename 'zzz.tar'

The tar file is downloaded and contains the files that were present in the folder.

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Technical debt
Tests only (no source changes)

Checklist:

Code changes
Unit tests added
Acceptance tests added
Documentation ticket raised:

update-docs · 2021-09-15T14:44:05Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

wkloucek · 2021-09-15T15:38:27Z

@jvillafanez this should also need some proxy configuration, so that oC Web can use this endpoint. Probably we also should add an capability @kulmann ?

ownclouders · 2021-09-15T15:47:27Z

💥 Acceptance tests Web-Tests-ocis-ocis-storage-1 failed. The build is cancelled...

butonic · 2021-09-15T16:58:57Z

there is already a capability, see https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/handlers/cloud/capabilities/capabilities.go#L105 and https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/data/capabilities.go#L110

but it seems we are not setting it https://github.com/owncloud/ocis/blob/master/storage/pkg/command/frontend.go#L66-L72 @jvillafanez you could try setting it in the filesCfg.

jvillafanez · 2021-09-15T17:11:22Z

there is already a capability, see https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/handlers/cloud/capabilities/capabilities.go#L105 and https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/data/capabilities.go#L110

Wrong context? The links point to "favorites", and I don't see any reference to the archiver there. The PR in reva implementing the archiver doesn't have any change around capabilities neither.

jvillafanez · 2021-09-16T10:51:38Z

Default route to the archiver added to the proxy.

$ curl -u admin:admin -k -OJ -v 'https://my.ocis.server/archiver/archiver?dir=/home/zzz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.0.2.27:443...
* TCP_NODELAY set
* Connected to my.ocis.server(10.0.2.27) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [836 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: O=Acme Corp; CN=OCIS
*  start date: Sep 16 09:46:40 2021 GMT
*  expire date: Sep 16 09:46:40 2022 GMT
*  issuer: O=Acme Corp; CN=OCIS
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
} [5 bytes data]
> GET /archiver/archiver?dir=/home/zzz HTTP/1.1
> Host: my.ocis.server
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.68.0
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [146 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Disposition: attachment; filename="zzz.tar"
< Content-Transfer-Encoding: binary
< Content-Type: application/octet-stream
< Date: Thu, 16 Sep 2021 09:56:51 GMT
< Vary: Origin
< X-Access-Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjMxODcyNjExLCJpYXQiOjE2MzE3ODYyMTEsImlzcyI6Imh0dHBzOi8vb2Npcy5qcC5zb2xpZGdlYXIucHJ2IiwidXNlciI6eyJpZCI6eyJpZHAiOiJodHRwczovL29jaXMuanAuc29saWRnZWFyLnBydiIsIm9wYXF1ZV9pZCI6ImRkYzIwMDRjLTA5NzctMTFlYi05ZDNmLWE3OTM4ODhjZDBmOCIsInR5cGUiOjF9LCJ1c2VybmFtZSI6ImFkbWluIiwibWFpbCI6ImFkbWluQGV4YW1wbGUub3JnIiwiZGlzcGxheV9uYW1lIjoiQWRtaW4iLCJ1aWRfbnVtYmVyIjoyMDAwNCwiZ2lkX251bWJlciI6MzAwMDB9LCJzY29wZSI6eyJ1c2VyIjp7InJlc291cmNlIjp7ImRlY29kZXIiOiJqc29uIiwidmFsdWUiOiJleUp3WVhSb0lqb2lMeUo5In0sInJvbGUiOjF9fX0.odeEizJVYCPzJX1uHhl-fmUxEx1NNahGNr_En3GoU7o
< Transfer-Encoding: chunked
< 
{ [5 bytes data]
100 3428k    0 3428k    0     0  5879k      0 --:--:-- --:--:-- --:--:-- 5890k
* Connection #0 to host my.ocis.server left intact
curl: Saved to filename 'zzz.tar'

butonic · 2021-09-17T10:24:35Z

there is already a capability, see https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/handlers/cloud/capabilities/capabilities.go#L105 and https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/data/capabilities.go#L110

Wrong context? The links point to "favorites", and I don't see any reference to the archiver there. The PR in reva implementing the archiver doesn't have any change around capabilities neither.

urgh yes ... sorry I was confused. I think a new files capability archives makes sense. But I would strongly prefer a url endpoint value that can be used to tell clients chere to go without having to hardcode it in the clients. For example

+							"archiver": map[string]interface{}{
+								"endpoint": "https://localhost:9200/archiver",
+								"max_num_files": 5000,
+								"max_size":      1000000,
+							},

butonic · 2021-09-17T10:59:36Z

storage/pkg/command/frontend.go

@@ -147,6 +147,13 @@ func frontendConfigFromStruct(c *cli.Context, cfg *config.Config, filesCfg map[s
 					"timeout":                86400,
 					"insecure":               true,
 				},
+				"archiver": map[string]interface{}{
+					"prefix":        cfg.Reva.Frontend.ArchiverPrefix,


If it is a path, then it is on the same host as the capabilities. But it could be on a dedicated host, eg https://archiver.owncloud.com.

Suggested change

"prefix": cfg.Reva.Frontend.ArchiverPrefix,

"url": cfg.Reva.Frontend.ArchiverURL,

there is already a capability, see https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/handlers/cloud/capabilities/capabilities.go#L105 and https://github.com/cs3org/reva/blob/master/internal/http/services/owncloud/ocs/data/capabilities.go#L110

Wrong context? The links point to "favorites", and I don't see any reference to the archiver there. The PR in reva implementing the archiver doesn't have any change around capabilities neither.

urgh yes ... sorry I was confused. I think a new files capability archives makes sense. But I would strongly prefer a url endpoint value that can be used to tell clients chere to go without having to hardcode it in the clients. For example

+ "archiver": map[string]interface{}{ + "endpoint": "https://localhost:9200/archiver", + "max_num_files": 5000, + "max_size": 1000000, + },

Probably the endpoint should be a relative path only? We get the backend URL from a config file already (at least in web client). This way you'd go for something like archiverUrl = config.server + capabilities.files.archiver.endpoint in the frontend? @kulmann

But it should be possible for the rest of the services (datagateway, appprovider, etc) to do the same... so I think we should prioritize consistency in this case.

We could add a comment to rise awareness or adjust the usage string in the command line, although we might need to adjust the strings of the rest of the services.

No preference from my side. Whatever is easier (not only for implementation, but also for admin who needs to configure it). In the web ui we have both situations already - e.g. the downloadURL comes as absolute URL form the backend while we do most API requests via relative URLS with the server URL from the config.

the capabilities, like the /.well-known/openid-configuration endpoint should allow clients to configure themselves. I agree with @jvillafanez regarding consistency. All endpoints should be URLs. That includes relative URLs like just a path: /archiver as well as https://otherhost.test/archiver.

This allows the server to configure clients to use custom implementations of services. All clients should be able to handle absolute and relative URLs.

The prefix is just a "namespace" for the handlers. There is no full URL in that case...

https://github.com/cs3org/reva/blob/e96fa872a8f569451d1d81d7ddefb6129c6dbd68/pkg/rhttp/rhttp.go#L204-L207

In the capabilities, you will be able to add a full URL, see #2529

wkloucek · 2021-09-21T09:25:45Z

capabilities will be tackled in cs3org/reva#2088

wkloucek · 2021-09-22T12:01:56Z

proxy/pkg/proxy/proxy.go

@@ -379,6 +383,10 @@ func defaultPolicies() []config.Policy {
 					Endpoint: "/signin/",
 					Backend:  "http://localhost:9130",
 				},
+				{
+					Endpoint: "/archiver",


@jvillafanez I made the proxy route also match /archiver so that we can use curl -u einstein:relativity -k 'https://localhost:9200/archiver?dir=/home/Foo' --output Foo.tar

That's ok for me, although I don't know if we have plans to add additional actions against the archiver service at some point.

For example, if we want to archive huge folder, we might want to provide async actions such as /archiver/async?dir=folder to create the request, /archiver/async/{id}/state to check the state (queued, archiving fileA, ready...), /archiver/async/{id} to download the archive if finished.
This allows more flexibility since we wouldn't need to touch the routing.

sonarcloud · 2021-09-22T12:05:51Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

100.0% Coverage
0.0% Duplication

wkloucek · 2021-09-23T08:34:08Z

storage/pkg/command/frontend.go

@@ -147,6 +147,13 @@ func frontendConfigFromStruct(c *cli.Context, cfg *config.Config, filesCfg map[s
 					"timeout":                86400,
 					"insecure":               true,
 				},
+				"archiver": map[string]interface{}{
+					"prefix":        cfg.Reva.Frontend.ArchiverPrefix,


The prefix is just a "namespace" for the handlers. There is no full URL in that case...

https://github.com/cs3org/reva/blob/e96fa872a8f569451d1d81d7ddefb6129c6dbd68/pkg/rhttp/rhttp.go#L204-L207

In the capabilities, you will be able to add a full URL, see #2529

Change the full URL is possible via Capabilities, not in the http handler in the archiver service

Merge: 255a6a2 889b30c Author: Willy Kloucek <[email protected]> Date: Thu Sep 23 10:37:44 2021 +0200 Merge pull request #2509 from owncloud/expose_reva_archiver Expose configuration for the reva archiver

phil-davis · 2021-11-09T08:47:45Z

Note for future readers. The API endpoint in cs3org/reva seems to be download_archive cs3org/reva#2066

The API endpoint here in oCIS is archiver

Expose configuration for the reva archiver

8844b0b

jvillafanez self-assigned this Sep 15, 2021

phil-davis mentioned this pull request Sep 15, 2021

Add API acceptance tests for /archiver endpoint #2510

Closed

Add changelog entry

075bd75

jvillafanez marked this pull request as ready for review September 15, 2021 15:33

jvillafanez requested review from wkloucek, refs and butonic September 15, 2021 15:34

Add default route to the archiver

c1dd0ea

butonic previously requested changes Sep 17, 2021

View reviewed changes

make archiver routes match also "/archiver" instead of "/archiver/" only

889b30c

wkloucek force-pushed the expose_reva_archiver branch from 93ac175 to 889b30c Compare September 22, 2021 11:53

wkloucek reviewed Sep 22, 2021

View reviewed changes

kulmann mentioned this pull request Sep 23, 2021

[full-ci] Initial implementation of archiver v2 owncloud/web#5832

Merged

14 tasks

wkloucek approved these changes Sep 23, 2021

View reviewed changes

wkloucek merged commit c5a7494 into master Sep 23, 2021

delete-merged-branch bot deleted the expose_reva_archiver branch September 23, 2021 08:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose configuration for the reva archiver #2509

Expose configuration for the reva archiver #2509

jvillafanez commented Sep 15, 2021

update-docs bot commented Sep 15, 2021

wkloucek commented Sep 15, 2021

ownclouders commented Sep 15, 2021

butonic commented Sep 15, 2021

jvillafanez commented Sep 15, 2021

jvillafanez commented Sep 16, 2021 •

edited

Loading

butonic commented Sep 17, 2021 •

edited

Loading

butonic Sep 17, 2021

pascalwengerter Sep 17, 2021 •

edited

Loading

jvillafanez Sep 17, 2021

kulmann Sep 17, 2021 •

edited

Loading

butonic Sep 17, 2021 •

edited

Loading

wkloucek Sep 23, 2021

wkloucek commented Sep 21, 2021

wkloucek Sep 22, 2021

jvillafanez Sep 22, 2021

sonarcloud bot commented Sep 22, 2021

wkloucek Sep 23, 2021

phil-davis commented Nov 9, 2021

	"prefix": cfg.Reva.Frontend.ArchiverPrefix,
	"url": cfg.Reva.Frontend.ArchiverURL,

Expose configuration for the reva archiver #2509

Expose configuration for the reva archiver #2509

Conversation

jvillafanez commented Sep 15, 2021

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

update-docs bot commented Sep 15, 2021

wkloucek commented Sep 15, 2021

ownclouders commented Sep 15, 2021

butonic commented Sep 15, 2021

jvillafanez commented Sep 15, 2021

jvillafanez commented Sep 16, 2021 • edited Loading

butonic commented Sep 17, 2021 • edited Loading

butonic Sep 17, 2021

Choose a reason for hiding this comment

pascalwengerter Sep 17, 2021 • edited Loading

Choose a reason for hiding this comment

jvillafanez Sep 17, 2021

Choose a reason for hiding this comment

kulmann Sep 17, 2021 • edited Loading

Choose a reason for hiding this comment

butonic Sep 17, 2021 • edited Loading

Choose a reason for hiding this comment

wkloucek Sep 23, 2021

Choose a reason for hiding this comment

wkloucek commented Sep 21, 2021

wkloucek Sep 22, 2021

Choose a reason for hiding this comment

jvillafanez Sep 22, 2021

Choose a reason for hiding this comment

sonarcloud bot commented Sep 22, 2021

wkloucek Sep 23, 2021

Choose a reason for hiding this comment

phil-davis commented Nov 9, 2021

jvillafanez commented Sep 16, 2021 •

edited

Loading

butonic commented Sep 17, 2021 •

edited

Loading

pascalwengerter Sep 17, 2021 •

edited

Loading

kulmann Sep 17, 2021 •

edited

Loading

butonic Sep 17, 2021 •

edited

Loading