Core Feature: Remote Storage Backends

[jpillora]

cloud-torrent is commonly run on cheap/free hosting providers which provide limited disk space and RAM though have access to large amounts of bandwidth. Considering this, it makes sense to support remote storage backends.

Required:

• Enhance download process to allow streaming (Minimising disk usage via TorrentDataOpener anacrolix/torrent#68)
• Use configurable download buffers to control how much can be downloaded ahead of the upload

Optional:

• Implement Dropbox backend (with auth/configuration in frontend)
• Implement Google Drive backend (with auth/configuration in frontend)
  • Implement OAuth2 support for third-party backend setup
• Implement HTTP client backend (stream torrent directly as HTTP download)
• Implement S3 backend (partially implemented https://github.com/ryansb/af3ro though missing multipart uploads), with auth/configuration in frontend
• Implement Mega backend (https://github.com/t3rm1n4l/go-mega)
• Implement Backblaze B2 backend (https://github.com/kothar/go-backblaze)
• Implement FTP,SFTP,SCP backends

[t-pankajkumar]

Hi jpillora,
I had written a java server application which takes the download url as input and uploads it to Dropbox of any size depends on the account storage, i had hosted it on Openshift.

please have a look at my code
https://github.com/t-pankajkumar/DropBoxJava/blob/master/DropBoxJava/src/com/drop/box/UrlUploadMain.java

i hope this will help you.
thank you :)

[joeblew99]

suggest you checkout minio on github. Its a S3 client in go, but also GS and its own parallel storage server.
it might be a better way to integrate with afero

[besoeasy]

Maybe A Option To Move Downloaded Torrent's To A FTP Server

[jpillora]

@besoeasy ah yep, will add that and SCP/SFTP to the list (though note, it will stream to the remote server, so there won't be a local copy)

[t-pankajkumar]

Here is a small code snippet to auto upload torrents to Openload i wrote a node js with express
`var express = require('express');
var WebTorrent = require('webtorrent');
var path = require("path");
var app = express();
var client = new WebTorrent();
var request = require('request');
var openload_api = 'your openload api';
var openload_key = 'your openload key';
var oUrl = 'https://api.openload.co/1/remotedl/add?login='+openload_api+'&key='+openload_key+'&url=';
app.use(express.static(__dirname + '/public'));

// views is directory for all template files
app.set('views', __dirname + '/views');
app.set('view engine', 'ejs');

//this provides download link for downloaded files
app.get('/download', function(req, res){
var file = path.join(__dirname,'public', req.query.file);
res.download(file); // Set disposition and send it.
});

//to add torrent enter 'your_heroku_name.herokuapp.com/torAdd?magnet=magnet_link
app.get('/torAdd', function(req, res){
console.log('started');
client.add(req.query.magnet, { path: 'public' }, function (torrent) {
torrent.on('done', function () {
console.log('torrent download finished');
torrent.files.forEach(function(file){
console.log(file.name+' '+file.length+' '+file.path+'\n');
request(oUrl+"https://your_heroku_name.herokuapp.com/download?file="+file.path, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body) // Show the HTML for the Google homepage.
}
});
});
});
});
res.send("downloading");
});`

[dubailive]

Hello, great script here. I created a Digital Ocean droplet and installed it works great. Just wondering if Cloud Torrent can be installed on Dedicated Server from a hosting company running Ubuntu 16.04 - advantage is to get much larger storage and unlimited bandwidth at cheaper price as well. instead of only using Digital Ocean or other cloud providers?

[sxml]

T-pankajkumar How to add this code? #

[t-pankajkumar]

#sxml just create a heroku simple nodejs app and use that script, i have been downloading torrents via my own site hosted at heroku :)
If you need i will share my code to you.

[cinetube]

@t-pankajkumar - please share your torrent site code.

[ajthemacboy]

Google Drive, Amazon Cloud Drive, and Dropbox don't allow certain... sensitive media to be hosted on their platforms. Basic encryption or obfuscation may be useful, but I think it would require a web frontend. Maybe something to consider for a future version.

[Kingkai2009]

hello t-pankajkumar can you show how to do in heroku with your code it will be very helpful for me

[teun95]

Would it not make sense to allow rclone to be used for this? At least for the time being. It already supports most of the backends mentioned here and supports encryption. The possibility to move files when they are done downloading and run post-processing scripts might even make integrating this unnecessary.

[diman82]

@teun95 You miss the point. Think if you download a 20GB file - you'll have to wait for it to dl to a local storage, and then and only then you may be able to sync it to some cloud. But the purpose of this important feature, is to stream the contents "uploading on the fly".

[teun95]

@diman82 I was not aware of that. That would be a first as far as I know as I have not seen that before at commercial seedbox services. I know that for some storage providers, rclone is able to upload streaming as long as the files are downloaded sequentially. This is used when mounting online storage. I am not sure how usable that is though for cloud-torrent.

But what you propose sounds like a lot of work that will take a long time and will not be available for many different cloud storage providers, let alone with encryption. Though even in a limited form it sounds like a really cool feature. Perhaps it would additionally make sense to also make using rclone (or any other custom script) possible as part of post-processing? Let's not forget that the storage issue of cheap VPS servers might become less of an issue pretty rapidly with declining storage prices.

[diman82]

@teun95 You're right in your concept, but your request can be quite easily achieved with a shell script, and it doesn't directly relates to the current project.
Simply monitor a folder and rclone will pick up any changes and copy/move content to a pre-configured cloud source.