[WIP] Datatx driver #1242

redblom · 2020-10-12T10:38:42Z

Datatx driver pkg setup.
note: Datatx interface methods' signatures still indefinite.

labkode · 2020-10-19T07:40:56Z

@redblom good progress!

redblom · 2020-10-28T12:55:14Z

This now implements rclone driver DoTransfer, GetTransferStatus.
There are also 2 cli commands available for testing this: rclone-do-transfer, rclone-get-transfer-status
For tests you'll need a running rclone server and an rconfig file with the remotes/users specified.
Transfer eg. a cli triggered transfer between two running reva instances:

$ ./cmd/reva/reva rclone-do-transfer -senderEndpoint http://rclone-reva:5572/ -srcEndpoint reva-1 -srcPath /test/1gb.txt -srcToken marie -destEndpoint reva-2 -destPath /myfiles/1gb1.txt -destToken marie

This will echo the rclone job id.
Notes:

src/destEndpoint - the remotes' names as specified in the config
the tokens are not used but required by the cli

Get status eg.:

$ ./cmd/reva/reva rclone-get-transfer-status -senderEndpoint http://rclone-reva:5572/ -transferID 68

This will echo the transfer status.

redblom · 2020-11-04T16:12:49Z

All 3 methods (DoTransfer, GetTransferStatus, CancelTransfer) of rclone driver implemented.

Use reva cli rclone transfer commands:
(do transfer)
./reva rclone-do-transfer -senderEndpoint http://rclone-reva:5572 -srcEndpoint reva -srcPath /test/test01.txt -srcToken marie -destEndpoint reva-2 -destPath /myfiles/02112020-01.txt -destToken marie

(get transfer status)
./reva rclone-get-transfer-status -senderEndpoint http://rclone-reva:5572 -transferID 1

(cancel transfer)
./reva rclone-cancel-transfer -senderEndpoint http://rclone-reva:5572 -transferID 1

ishank011

The driver is supposed to be included in the GRPC service, not invoked directly by clients. Please take a look at the other GRPC services and make sure that this also follows that approach https://github.com/cs3org/reva/blob/master/internal/grpc/services/userprovider/userprovider.go

redblom · 2020-11-11T07:46:23Z

@ishank011 yeah I know, I'm working on those right now. These rclone clients are for driver testing purposes only, I don't expect them to be part of the final PR.

cmd/reva/main.go

cmd/reva/transfer-create.go

internal/grpc/services/datatx/datatx.go

pkg/datatx/driver/rclone/rclone.go

pkg/datatx/driver/config/config.go

pkg/datatx/driver/txdriver.go

pkg/datatx/driver/rclone/rclone.go

ishank011

Added a few comments. I'm not sure I fully understand how the token situation works. In OCM we transfer a token when sharing a file which is used to access resources on the file owner's provider, not sure how that translates here and how it connects to that, since these will be enabled only once an OCM share has been created, right? (https://wiki.cs3mesh4eosc.eu/wps/4/task44stuff/apis)

ishank011 · 2021-01-05T11:38:13Z

pkg/datatx/loader/loader.go

@@ -0,0 +1,25 @@
+// Copyright 2018-2020 CERN


Create a folder called manager and move the loader, registry and rclone folders inside it. Refer how we do it in other packages.

ishank011 · 2021-01-05T11:39:28Z

internal/grpc/services/datatx/datatx.go

@@ -33,14 +39,18 @@ func init() {
 	rgrpc.Register("datatx", New)
 }

-type config struct {
+func (c *config) init() {
+	// set sane defaults


Set default value for Driver to rclone

ishank011 · 2021-01-05T11:46:35Z

internal/grpc/services/datatx/datatx.go

+	// (token is username for now, it is used like that in the current rclone driver implementation)
+	// TODO: re-implement when the connection-string-remote syntax is available in rclone
+	// TODO: implement proper token use
+	srcToken := u.GetUsername()


How do these tokens work? For the source token, we use the username and for destination, we use the grantee's ID?

Yeah, the destination token should be available here, returned by OCM share request. But this is not implemented yet, see below: #1242 (comment).
Apparently a source token for the 'local' source file to be transferred should (at this point) be created somehow (see https://wiki.cs3mesh4eosc.eu/wps/4/task44stuff/apis#data-transfer 'generate token to read Hugo's file'). Maybe you could shed some light on (how to do) that.

So yes, the driver.DoTransfer token parameters are being 'abused' here. We need actual tokens.

ishank011 · 2021-01-05T11:47:02Z

internal/grpc/services/datatx/datatx.go

+	txID := req.TxId.OpaqueId
+
+	// TODO find the jobID belonging to the transfer id; let transfer id be the jobID for now
+	var jobID int64


No need to declare jobID before initializing

ishank011 · 2021-01-05T12:22:54Z

pkg/datatx/rclone/rclone.go

+}
+
+// rcloneCancelTransferReqJSON the rclone job/stop method json request
+type rcloneCancelTransferReqJSON struct {


Can we use a generic struct for rcloneCancelTransferReqJSON and rcloneStatusReqJSON, since they have the same fields? (same for rcloneCancelTransferResJSON)

I will declare them within the function there used in, that's actually what I wanted but thought it was invalid to declare a struct inside a method (my lack of golang knowledge).

redblom · 2021-01-06T11:57:37Z

Added a few comments. I'm not sure I fully understand how the token situation works. In OCM we transfer a token when sharing a file which is used to access resources on the file owner's provider, not sure how that translates here and how it connects to that, since these will be enabled only once an OCM share has been created, right? (https://wiki.cs3mesh4eosc.eu/wps/4/task44stuff/apis)

With data transfer a special transfer OCM share request (which will be a to be defined special datatx OCM protocol type) is done towards the anticipated destination and at destination a token is created and returned so the token is available at source site for use when doing the actual transfers (transfers are instigated at the source site so it must have a valid token).
Actually this is not described in the notes in CreateTransfer method of grpc service datatx.go. I will do that; should make it clear.

…atus

…es modified.

ishank011

Persistency should be implemented in the same interface, there shouldn't be the need for a new one. For example, refer how we do it for OCM invites https://github.com/cs3org/reva/blob/master/pkg/ocm/invite/manager/json/json.go

redblom · 2021-03-02T13:02:28Z

When the (datatx) share is received in the ocmd we know it's a datatx share through the protocol options parameter. However we currently cannot persist it as such. Share objects have no tx specifics. So the receiver does not know it's a datatx share either. How do we go about that?

ishank011 · 2021-03-02T13:41:56Z

Why can't we persist it? On the recipient side, the request would look like as described in https://wiki.cs3mesh4eosc.eu/wps/4/task44stuff/apis, so it would know it's a datatx share.

redblom · 2021-03-02T14:05:51Z

The datatx transfer ends up as a 'marshalled' Share object in the received_shares section of the shares file at the receiver's end. What options do I have to define there that it is a datatx share; so that the receiver (and the system) when accepting this share know that after reading this share from the shares file that this is a datatx share and the transfer can commence?

In other words, in the shares file, how does a datatx share differ from a regular share?

ishank011 · 2021-03-02T14:24:49Z

Add one more parameter to the schema?

redblom · 2021-03-02T14:42:48Z

That is one solution I was also thinking of.

So simplest:

cs3.sharing.ocm.v1beta1.Share { 
   ... ;
   bool is_data_transfer_share = 10;
}

Or maybe share_type in case there there could be other future types, don't know, maybe stream type ...

ishank011 · 2021-03-02T16:13:13Z

Sure, share_type would be better. Please create a PR and in the description part in the proto file, mention this use case

redblom · 2021-03-03T10:43:25Z

cs3org/cs3apis#109

redblom · 2021-03-04T11:51:45Z

How about using the share ID instead of transfer ID in cs3 datatx api. The share ID is already there. That would simplify things.

redblom · 2021-03-12T11:01:59Z

@ishank011 : I believe pkg.ocm.share manager interface method Save() needs an additional share type arg.

ishank011 · 2021-03-12T11:06:58Z

@redblom sure, feel free to make changes as you need.

redblom · 2021-03-23T09:15:21Z

@ishank011 please review #1578

redblom · 2021-03-24T16:49:06Z

@ishank011 : implementing this (#1578) I'm thinking we should not use CreateTransferRequest at all at the origin's end for a transfer but use CreateOCMShareRequest directly. An OCM share will always be the starting point anyway. So using reva cmd ocm-share-create would look something like:

cmd/reva/reva -insecure ocm-share-create -shareType transfer -grantee jarek1234 -granteeType user -idp softwaremind.com /home/test

I think it makes more sense.

Then we change the CS3 datatx CreateTransferRequest message to contain Share (or at the minimum the share id) (and optional opaque) fields only. *Note that the actual share id will be the transfer identifier.

Now when UpdateReceivedOCMShare is called at the recipient's end the CreateTransferRequest can follow with the share specified. And then the actual transfer can be initiated with a driver call:

pkg.datatx.TxDriver.CreateTransfer(shareID, ...src..., ...dest...)

TODO: the transfer driver needs a destination path. In case this is user defined it should be specified in the CreateTransferRequest message as well. The user interaction needed for this needs to be taken into account in the design.
2 options:
. through CreateTransferRequest by extending this message with a destination path
. through UpdateReceivedOCMShare and persist the destination path together with the share (Share needs to be extended). CreateTransferRequest does not need to be extended with destination path because it will be part of the persisted share.

ishank011 · 2021-03-26T12:08:27Z

When UpdateReceivedOCMShare is called, the reference is created in the shares directory https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/ocmshareprovider.go#L260 -> https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/ocmshareprovider.go#L291-L349, where the default path is /home/Shares/name-of-shared-resource. We can use the same path as the destination.

For WebDAV share references, we have the target webdav://token@idp?name=share-name. For transfers, we don't really need a target as the whole resource will be located here and we can just pass this path to rclone.

These references are statted in the gateway storageprovider service https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/storageprovider.go#L1369-L1395. We'll need to add a case for transfer refs to just be returned as it is. We have all the existing functionalities that will allow users to make changes to any files/folders located inside this reference.

redblom · 2021-03-30T11:39:24Z

@ishank011 I'd like to see this merged first (create transfer type OCM share) #1578

redblom · 2021-04-08T07:22:13Z

When UpdateReceivedOCMShare is called, the reference is created in the shares directory https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/ocmshareprovider.go#L260 -> https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/ocmshareprovider.go#L291-L349, where the default path is /home/Shares/name-of-shared-resource. We can use the same path as the destination.

@ishank011 What would be the resulting location here, eg. marie/name-of-shared-resource ? And then ./reva stat /home/Shares/name-of-shared-resource would work?

ishank011 · 2021-04-08T12:16:16Z

You're passing the /remote.php/webdav endpoints for both PUT and GET to rclone, right? Then you just need to use the path /home/Shares/name-of-shared-resource everywhere. Reva will resolve that based on the user in context.

redblom · 2021-04-08T12:56:54Z

You're passing the /remote.php/webdav endpoints for both PUT and GET to rclone, right? Then you just need to use the path /home/Shares/name-of-shared-resource everywhere. Reva will resolve that based on the user in context.

@ishank011 I use single endpoints for each request. For transfer the rclone service is called through 2 possible endpoints, operations/copyfile to transfer a file and sync/copy to transfer a folder structure. These calls will asynchronous and immediately return a jobID (to be used for status/cancel requests).
The requests' data is in the form of a single string with all necessary parameters included (https://rclone.org/docs/#connection-strings). A folder transfer example of such a req data string would be:
'{"_async":true,"srcFs":":webdav,bearer_token=\"...token...\",url=\"http://reva-datatx:20080/webdav\":/big-data","dstFs":":webdav,bearer_token=\"...token...\",url=\"http://reva-2:20080/webdav\":/big-data"}'
This would be a transfer to a 'local' path, eg marie/big-data/...

I experimented with accepted shares where I gave rclone the share path like url=\"http://reva-2:20080/webdav\":/MyShares/big-data but that did not quite work. I was expecting these files to exist somewhere in (my case) marie/big-data/... but for the user at /MyShares/big-data/.... So first I want to know if this is indeed what's expected.

ishank011 · 2021-04-08T13:44:54Z

Which service handles the requests at the /webdav endpoint?

redblom · 2021-04-08T14:11:09Z

Which service handles the requests at the /webdav endpoint?

../../internal/http/services/owncloud/ocdav/ocdav.go

@ishank011 I'm also not sure what should be returned here for the transfer case (https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/storageprovider.go#L1369-L1395) exactly? Simply the provided resourceInfo? How should we establish here that this is a transfer ref?

ishank011 · 2021-04-08T15:04:02Z

Use the path /remote.php/webdav/home/MyShares/xyz. It should work

Simply the provided resourceInfo?

Yes

How should we establish here that this is a transfer ref?

Check the scheme

redblom · 2021-04-08T15:31:09Z

Use the path /remote.php/webdav/home/MyShares/xyz. It should work

Yes, that seems to work!

Simply the provided resourceInfo?

Yes

How should we establish here that this is a transfer ref?

Check the scheme

And that should read datatx I suppose, because now it's webdav. That makes sense.

redblom · 2021-04-12T12:43:42Z

Simply the provided resourceInfo?
Yes

How should we establish here that this is a transfer ref?

Check the scheme

And that should read datatx I suppose, because now it's webdav. That makes sense.

@ishank011 : I can set the scheme to datatx here:

reva/internal/grpc/services/gateway/ocmshareprovider.go

Line 324 in 1765ae8

    
           TargetUri: fmt.Sprintf("webdav://%s@%s?name=%s", token, share.Creator.Idp, share.Name),

Then I can determine it at https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/storageprovider.go#L1369-L1395 for the new case. Is that correct?

redblom · 2021-12-13T09:56:43Z

This PR is superseded by #2052

redblom requested a review from labkode as a code owner October 12, 2020 10:38

labkode previously approved these changes Oct 19, 2020

View reviewed changes

redblom dismissed labkode’s stale review via fac695b October 28, 2020 12:09

redblom force-pushed the datatx-driver branch from ea6aaf6 to fac695b Compare October 28, 2020 12:09

redblom force-pushed the datatx-driver branch from fac695b to 7e1d7b1 Compare November 4, 2020 15:49

ishank011 suggested changes Nov 10, 2020

View reviewed changes

redblom force-pushed the datatx-driver branch from 7e1d7b1 to 81ef606 Compare November 12, 2020 15:22

ishank011 suggested changes Nov 30, 2020

View reviewed changes

redblom force-pushed the datatx-driver branch from 81ef606 to e12d56e Compare December 9, 2020 14:14

ishank011 reviewed Jan 5, 2021

View reviewed changes

redblom force-pushed the datatx-driver branch from e91cf70 to bfcf363 Compare January 7, 2021 10:53

Antoon Prins added 14 commits January 7, 2021 16:52

Setup for datatx driver pkg.

382635a

Changelog for datatx driver pkg.

d1fc3d4

PR link added.

ac750c5

2 new rclone cli commands: rclone-do-transfer, rclone-get-transfer-st…

7d0e86a

…atus

err check added.

77c1397

loader, config added.

239a2c7

Status consts, Job, Remote structs added. Interface methods' signatur…

b9ad3f6

…es modified.

DoTransfer, GetTransferStatus implemented.

8471092

Cli command for cancel status.

31876cb

Small fixes.

85586b7

Return type of CancelTransfer changed to string.

d9bd092

CancelTransfer implemented.

002ace7

Config changed. txdriver.Job removed.

b11ce1d

Config changed. txdriver.Job removed.

150acaf

New config flags added.

c2c82e2

ishank011 suggested changes Jan 15, 2021

View reviewed changes

ishank011 force-pushed the master branch from 71c6a47 to 48f10db Compare February 3, 2021 14:23

redblom mentioned this pull request Mar 3, 2021

Share type added to specify Share purpose. cs3org/cs3apis#109

Merged

This was referenced Mar 17, 2021

Implement cs3.sharing.collaboration.v1beta1.Share.ShareType #1563

Merged

Datatx pull model createtransfer #1578

Closed

redblom closed this Dec 13, 2021

[WIP] Datatx driver #1242

[WIP] Datatx driver #1242

Conversation

redblom commented Oct 12, 2020 • edited Loading

labkode commented Oct 19, 2020

redblom commented Oct 28, 2020

redblom commented Nov 4, 2020 • edited Loading

ishank011 left a comment

Choose a reason for hiding this comment

redblom commented Nov 11, 2020

ishank011 left a comment

Choose a reason for hiding this comment

ishank011 Jan 5, 2021

Choose a reason for hiding this comment

ishank011 Jan 5, 2021

Choose a reason for hiding this comment

ishank011 Jan 5, 2021

Choose a reason for hiding this comment

redblom Jan 6, 2021

Choose a reason for hiding this comment

ishank011 Jan 5, 2021

Choose a reason for hiding this comment

ishank011 Jan 5, 2021

Choose a reason for hiding this comment

redblom Jan 7, 2021

Choose a reason for hiding this comment

redblom commented Jan 6, 2021

ishank011 left a comment

Choose a reason for hiding this comment

redblom commented Mar 2, 2021

ishank011 commented Mar 2, 2021

redblom commented Mar 2, 2021

ishank011 commented Mar 2, 2021

redblom commented Mar 2, 2021 • edited Loading

ishank011 commented Mar 2, 2021

redblom commented Mar 3, 2021

redblom commented Mar 4, 2021

redblom commented Mar 12, 2021 • edited Loading

ishank011 commented Mar 12, 2021

redblom commented Mar 23, 2021

redblom commented Mar 24, 2021 • edited Loading

ishank011 commented Mar 26, 2021

redblom commented Mar 30, 2021

redblom commented Apr 8, 2021

ishank011 commented Apr 8, 2021 • edited Loading

redblom commented Apr 8, 2021 • edited Loading

ishank011 commented Apr 8, 2021

redblom commented Apr 8, 2021

ishank011 commented Apr 8, 2021

redblom commented Apr 8, 2021

redblom commented Apr 12, 2021

redblom commented Dec 13, 2021

redblom commented Oct 12, 2020 •

edited

Loading

redblom commented Nov 4, 2020 •

edited

Loading

redblom commented Mar 2, 2021 •

edited

Loading

redblom commented Mar 12, 2021 •

edited

Loading

redblom commented Mar 24, 2021 •

edited

Loading

ishank011 commented Apr 8, 2021 •

edited

Loading

redblom commented Apr 8, 2021 •

edited

Loading