Use Amazon Cloud Drive as external storage

[wdenx]

It would be nice if we could use Amazon Cloud Drive (ACD) as external storage,
for obvious reasons ideally with automatic encryption of the data.
ACD looks interesting as it offers really cheap storage (unlimited storage for < $60 / year).
Note that ACD is not compatible with S3, it uses a different API.

Code to access this API exists (see for example https://github.com/yadayada/acd_cli )

Thanks in advance,

Wolfgang

[DeepDiver1975]

NOTE: starting with 8.2 we will open up the files_external APIs even more to allow any developer to implement external filesystems as an app - anybody will then be in the position to implement external storages like this.

[ronoliverclarin]

Any updates on this one?
Or Should I say any developer working on this one?

[LukasReschke]

Any updates on this one?
Or Should I say any developer working on this one?

No.

[masteranalyze]

👍 it`s an good ideea,and i hope this gets implemented.

Amazon Cloud Drive (ACD) as external storage is definetely worth to be implemented into owncloud!

And like deepdiver said,it will be also nice,that if others appears to be able to easily implement them,into owncloud,to have alternatives storages.

1)I did not manage to test this : automatic encryption of the data,but tell me something,if encryption of the data is used,practically if we have 10.000 mp3 files song files,amazon,will not be able to playback that music?It will not know that it`s an mp3 file ?And the file will be able to be playback in OWNCLOUD?

2)Or it`s just an encryption on sending the data between amazon cloud drive and owncloud?

If somebody can clear me out,about this feature,it will be real nice.Most good,it will be 1st,so you can have an total control of the privacy of the files that you store into the 3rd party clouds.

[RobinMcCorkell]

This can be implemented as an app, especially with the new Backend API

[AndrewBucklin]

Am I to assume that this was closed because it won't be implemented as a storage option for owncloud?

[RobinMcCorkell]

@AndrewBucklin While it won't be implemented in the near future within ownCloud core, you are free to make an app that implements the required interfaces 😄

[Storycrafter]

High personal interest = I'll be investigating this and gathering an idea of the effort required for me to implement as an app. I'll comment back one way or another in a day or three.

[dellipse]

What did you find out, @Storycrafter ? Any luck?

[Storycrafter]

It's not as trivial as I had hoped. The documentation around the ownCloud storage mechanism is limited, and I'm by no means an expert at this point. I'm still keenly interested. Storage Made Easy has become a horrible company to work with for some of my projects so I need a solution I can control now more than ever.

At this point I can report that I see no reason it can not be done, but it also would not be a weekend project for me. I'll put together a task list and some tech spec type docs and post them somewhere.

[masteranalyze]

Maybe @Xenopathic or @DeepDiver1975 can help with some proper documentation,and make things easy at least if the core team does not want to do this,at least leave others to do it and provide proper documentation so people with skills like storycrafter to be able to help the whole community.

At least to easily the job of storycrafter,storage made easy is crap and you must get rid of that my friend.

[wdenx]

Maybe it would help to have a look at "A command line interface and FUSE filesystem for Amazon Cloud Drive" (acd_cli) implementation at https://github.com/yadayada/acd_cli ?
These tools work really well for me, and response to problems/questions has been excellent.

[Storycrafter]

Thanks @wdenx, I'm aware of that plus I'd looked at the Cloud API prior to looking around for ways to integrate. I'm actually more concerned about the OC side of things, although after posting a little earlier and digging at the code again today with fresh eyes, I feel like I picked up a few clues. I'm going to set up a dev environment sometime this week and start hacking.

[JohnnyCalavera]

It could be an even better idea to create an app doing "rsync for cloud storage[s]" in general (not limited to acd).
So to speak a rclone app (http://rclone.org/)
You'd get:

• Google Drive
• Amazon S3
• Openstack Swift / Rackspace cloud files / Memset Memstore
• Dropbox
• Google Cloud Storage
• Amazon Cloud Drive
• Microsoft One Drive
• Hubic
• Backblaze B2
• Yandex

[wdenx]

Dear Johnny,

In message owncloud/core/issues/17012/[email protected] you wrote:

It could be an even better idea to create an app doing "rsync for cloud storage[s]" in general (not limited to acd).

Well, the acd_cli based approach to mount the ACD as a FUSE file
system under Linux is working pretty well for me; but you probably
don't think of Linux based environments only, right?

Best regards,

Wolfgang Denk

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Two wrongs don't make a right, but three rights make a left.

[JohnnyCalavera]

but you probably don't think of Linux based environments only, right?

Dear Wolfgang,

as a matter of fact I do think of Linux based environments - especially the Rockstor NAS OS (CentOS-based). Nonetheless the rclone approach would give us more options in terms of cloud service providers and since it is still actively developed there will be even more possibilities in the future.
So the owncloud app could end up being a rather simple one and it's code doesn't need to be touched that often because the underlying rclone software will perform the syncing tasks while the owncloud app is barely more than a GUI for rclone.

Well, the acd_cli based approach to mount the ACD as a FUSE file system under Linux is working pretty well for me

I did test acd_cli a few weeks ago and it seemed to work pretty well. My only reason to bring rclone to the table is the fact that we would get way more possibilities without having to code a lot from scratch.

[Storycrafter]

Does anyone have a development environment set up by chance and mind throwing me a least-privilege login? I want to get started, but haven't found the time to properly set one up on top of other current priorities. Otherwise, it may be another week or two before I can get to this. Alternatively if anyone knows of an image I can put on a VPS or Virtualbox to get a dev environment quicker, that'd be welcomed.

[girvo]

@Storycrafter I'm about to re-setup my OwnCloud instance and am looking at backends for storage, so I'm likely to use ACD. Do you mind contacting me via email? I can provide an API key/account details/dev env perhaps :)

[Storycrafter]

@girvo ,thanks for the offer, but I was able to get a dev environment started at openshift. I plan to start taking a whack at things this weekend. I'll ping you via email as well.

[blubitv]

Hello guys,

did anyone manage to get a working solution and use ACD as external storage? (with rclone or acdfuse or some other connector)

[Orderchaos]

@blubitv Hey, not sure if you're even still interested in this, but I saw your question so figured better late than never.

I'm not a developer (although I'm considering looking into this just out of personal interest/hobby), but I do have an owncloud instance setup on a VPS with Amazon Cloud Drive mounted using acd_cli and it's been working well for several months now.

I used the acd_cli to mount it as a drive and then setup the external storage in owncloud (connecting to the ACD as 'local'). I've even shared access to owncloud with a few people and limited their storage to specific folders ('group' permission) to subfolders on a single amazon cloud drive account and it works exactly as expected. Here's a pic of my owncloud settings after getting acd_cli installed and mounted (see the github link in my previous paragraph for instructions on that):

[masteranalyze]

@Orderchaos did you done this in owncloud 9.01?

Can you replicate the steps for it?So i can test this.

In amazon cloud drive,i noticed,that when i wish to get my backup of arhives of an website,witch is basically over 10gb,so let`s say an backup file of 15gb,the shared link from amazon cloud drive,does not contain that /temp / path ,and the link is basically from amazon s3,and files of over 10gb mainly cannot be Wget directly into the linux server.
Somebody else did encountered this error ?It basically gives 403 error,because the link is from Amazon S3,not from amazon cloud drive,but the files are on amazon cloud drive.

I talked 100 times with the Amazonians,and i explained to them this problem,but they all seems to not understand,that the files are on Amazon cloud drive,they are asking for Amazon S3 account,but i don`t have no amazon S3,i have amazon cloud drive.I did even make them video,and they did not understand,again they asked for my amazon s3.

How the heck they hire people on Amazon?!?!I just wonder!

[masteranalyze]

@Orderchaos in acd_cli i saw : workaround for download of files larger than 10 GiB ,did you try to upload an 15gb file,and then download ?And can you tell us if this works or not ?I don`t understand what they are trying to mean with "WORKAROUND For files larger then 10gb",if anybody nows what this means,feel free to reply.

I know amazon haves limit for upload of 50gb for an file,but on download file limit,they are saying there is no limit,then why the heck files of over 10gb cannot be download in linux,centos or any other.

It can only be downloaded via Windows,with their Download Manager,but on linux seems to not work to download files of over 10gb.

And basically,if you can`t download files of over 10gb,why they let you upload files of up to 50gb?Because it does not make no sense.

You upload an 20gb file,and then if you need it,what you do?Your stuck on Windows ?!

[Orderchaos]

@masteranalyze larger files can timeout and I've had some issues with them causing sync problems when they do. I haven't tried a lot of large files with owncloud, but on a different server (same cloud drive) I've been able to use rsync to copy directly from a local drive to a mounted cloud drive with large media files (up to ~30GB). Just make sure that the acd_cli sync command doesn't run while the file is being transferred as that seems to be what causes the connection issues. Also don't have more than one transfer per server going at a time (can work, but can also cause more sync issues).

That's all for uploading though. Downloading I haven't had any problems with as I'm mostly using this for backup. Let me get a download started and see how it fares and I'll let you know.

Oh and this is on owncloud 8.2.2. I haven't updated in a while.

[masteranalyze]

@Orderchaos on upload on my side,i did not have no issues,because basically,i did not upload no file larger then 50gb,but the problem on my side,is on downloading files of over 10gb to linux.

There is no problem to download on windows with their download manager,an file of 15gb,but if you want to upload it directly into an webserver with linux with wget command,it gives error 403,i checked why is giving 403,because 403 means ACCESS denied,and when i used an firefox extension to give me the exactly link for Wget on Windows,the link of larger files of over 10gb,are links from AMAZON S3.

From what i know,amazon cloud drive is build up on amazon s3,as an retail version of the amazon s3,but i don`t know what the heck they are doing and how is their arhitecture exactly,for files of over 10gb.

Because if you link of the link shared for files of over 10gb,you will notice,is not anymore that /TMP/ link,witch is provided for all files of under 10gb,its an S3 link basically,and when you try to get that link witch is basically from Amazon S3,with wget on centos 6.7,the server gives access denied,and is kind of logical to give access denied,because i dont put no user or password on it or access key,basically i dont have no Amazon S3,but the files of over 10gb,seems to be there,and cant be donwnload in Linux.

Please try out with an single file of 15gb,and let me know.

Your using 64 bit system or 32bit system?Just checked out the documentation,and seems owncloud,also haves some restriction on uploading files larger then 2gb.I guess because of the browser.

I did not find out,what is the maximum file upload limit in owncloud,if anybody knows this?Please write your comment,in 64bit systems.

[Orderchaos]

@masteranalyze

Oh I see what you mean. The issue you're having is because you want a direct download link which Amazon doesn't provide (not in a straightforward manner anyways). There are ways to get it, but really you'd be better off doing it the way I mentioned which is mounting it as a network drive and using rsync or another copy method to copy the file from your amazon cloud drive to a local drive.

I just tested that and it worked fine for > 20GB file.

Also owncloud can be configured with custom upload limits. Mine is currently 10GB/file but I haven't tested anything quite that large yet.

[masteranalyze]

@Orderchaos thanks for testing,is good if it`s working like that,i will try it out,hopefully via owncloud,i will be able to download my backup directly into the linux server,and not downloading back in windows,and then re-uploading back in the linux server,because that is waste of time.

Any ideea about the file limits upload in owncloud as setup?On 64 bits,i cannot find nothing exactly into their documentation.

[masteranalyze]

@Orderchaos i just saw now ,that you mentioned that your is 10gb/file,10gb is fine.

What is the largest file gb you uploaded into owncloud till now?!via browser

[Orderchaos]

@masteranalyze here's the owncloud documentation on it: https://doc.owncloud.org/server/9.0/admin_manual/configuration_files/big_file_upload_configuration.html

I'll start an upload of a large file via owncloud and let you know how it handles.

[masteranalyze]

@Orderchaos i saw that,but it`s only mentioning this : " 64-bit filesystems have much higher limits; consult the documentation for your filesystem."

This are for 32bits systems not for 64bits systems:

< 2GB on 32Bit OS-architecture
< 2GB on Windows (32Bit and 64Bit)
< 2GB with Server Version 4.5 or older
< 2GB with IE6 - IE8
< 4GB with IE9 - IE11

"consult the documentation for your filesystem" this reffers to the filesystem on witch owncloud is run?So in my case it will be centos 6.7?!Or the filesystem of the external storage used?

Because what is in here : https://access.redhat.com/solutions/1532 ,i think as centos 6.x copies practically red hat,that means in ext4,16tb is the file upload limit?

From what i know till now from my research,only google drive can handle the largest upload size,witch is 500gb/per file.

Anyhow i dont think somebody needs 500gb per file,but 20-30gb if its working fine,it will be allright!

As for example upload limit in amazon cloud drive we know it s 50gb,so even if you put an limit of 60gb per file,is no use,because you wont get it into your amazon cloud drive.

Please let me know the filesize and how it`s handled!

[Orderchaos]

@masteranalyze those are theoretical limits based on the math behind 32 and 64bit addressing...Actual hard limits are dependent on the OS, but are still very likely higher than you will actually need.

If you've already got the server, why don't you just get it setup and try it out? Amazon has a free trial for the cloud drive and owncloud and the acd_cli are both free.

I'm currently uploading a ~5GB file via the owncloud web interface, but my upload speed isn't very fast so it may be a while. I will let you know if it has any problems.

[masteranalyze]

@Orderchaos i will deploy owncloud 9.0.2,in 5 may-6may,when they will release that update,i expect some fixes,witch should be in this release,and i will definetly try out everything you said,and share results here.

They are dependent on Os,not on the external file share you use or browser limits? Because if the Os permits 2TB file upload,and browser cannot handle that file,it will fail,same if amazon cloud drive limit is 50gb per file,is no use to set that limit to 50gb per file,maybe 20-30gb,to not be problems.

Did you tryed out if you already have setup owncloud on your side,for safety and privacy,the encryption,i mean if you have conected your amazon cloud drive into your owncloud with encryption enable,and basically you upload your pcbackup.zip file witch is 1gb your personal files,with encryption on,amazon will know what you have in your archive or how will that file appear to them?

Let`s say photo.jpg ,encrypted when upload via owncloud,stored in amazon cloud drive.

[Storycrafter]

I have a development environment setup and have identified the integration point in code. I've also submitted my signed agreement for contributor licensing -- no response on that (and not expected any time soon, honestly). I will track progress at http://mark-martin.com starting tomorrow.

[DeepDiver1975]

Hi Martin. Following your blog you are working on a fork of core. Please don't walk this path but better implement an app which provides the acd storage.
Please reach out on IRC in case you have questions.

Regarding your cla: i was answering you last night.

[Storycrafter]

@DeepDiver1975

Thanks for taking a look. When you suggest not to fork, are you suggesting to wait for commit privs and commit to a branch on the original report?

Others also mentioned going with an app instead of via files_external before. What is the benefit to that approach? Distribution? Are you suggesting that making it part of core isn't desirable?

[DeepDiver1975]

The core repo already holds a hell of code and especially all the external file systems put some burden on us with respect to unit testing. When implementing an external filesystem as an app on it's own it can live isolated in a repo of it's own where unit testing an integration testing can be setup much easier using travis ci.

As an example: https://github.com/icewind1991/files_external_ftp

[Storycrafter]

I think I understand now. The full API is available externally as an app, but as you suggest -- it can be isolated for testing and management purposes.

Would certainly make keeping the repo in synch with the upstream a ton easier if there is no upstream.

I'll modify my approach. Thanks for adding clarity.

[Storycrafter]

OK, I've restarted the bootstrapping and have taken the files_external_ftp as a starting template.

I'm encountering an issue on travis-ci with the core before_install.sh script. It fails to install oc core due to a cryptic error. If anyone has insights here, I'd appreciate hearing them.

https://travis-ci.org/Storycrafter/files_external_acd/jobs/140862133

[Flux3PO]

I don't own a VPS, can I run this on a Webspace only Install ?

I really want ACD to work as external Storage Provider.

[masteranalyze]

It seems people from owncloud ,are now NEXT CLOUD.

@Storycrafter check this link maybe it does help you to do the amazon cloud drive implementation 👍
https://help.nextcloud.com/t/amazon-cloud-drive-joomla-nextcloud-how-to/4607/6

[WLCD]

Any new development on this matter ?

[Flux3PO]

Yeah, would love to see this implemented as well

[philjak]

+1

[masteranalyze]

It seems owncloud it`s splited between old owncloud and nextcloud,i think this should be post in nextcloud 12 overview : nextcloud/server#2672 ,as nextcloud is FOSS complectly and owncloud is not FOSS on everything witch is an must at least on the security side.

I will ask there if there is any interest,and you can follow up,the more will be,maybe this gets implemented,as right now it seems dead.

[Flux3PO]

@masteranalyze
can you share the Link ? I've already ditched CM over NC and I only can advise everyone else doing the same.

[masteranalyze]

The link was in my prevision post,anyhow this is the link to NC 12 overview 👍
nextcloud/server#2672

But from what i see,nobody wants to implement this either if we talk about the old group of owncloud or the new cloud members,but maybe if we post more there,and ask for this,maybe things happens.

[rafaelsaru]

@Orderchaos I'm getting permissions errors when trying to add the Local storage to Owncloud.
Maybe the folder permissions is for root only and I can't change it by chown -R

[rafaelsaru]

I'm using acd_cli mount --allow-other -i 0 /mnt/

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.