Integrate Syncthing with Nextcloud and Owncloud. #8384
Comments
sunjam
changed the title
|
This can be done right now. Syncthing needs to be run as www-data (or more specifically same user as the webserver). A few caveats. Syncthing will not utilize the Nextcloud trash or versions folder by default. That being said one can set the custom folder path of Syncthing versioning to be the same as Nextcloud trash (nextcloud/data/user/files_trashbin). Secondly changes made by Syncthing will not be picked up in Nextcloud folders by default. One either has to run the nextcloud command
but this takes far to long to do this on a regular minute by minute basis for large folders or set all folders shared with Syncthing as external local folders in Nextcloud which will cause changes made by Syncthing to be picked up immediately in Nextcloud. Theoretically the config setting below can also be used (0 is default) instead of using external local folders but have not tried or tested this:
External local folders integrating Syncthing works great. Highly recommend. |
|
From all the long discussions I don't see any benefits. Now I don't know syncthing well, I had a look at the site - it is like btsync, syncing files between systems?W What kind of integration do you mean? How would it help with 'deltasync and LANsync'? You really should be a bit more elaborate, I don't get what this is meant to do and that is probably why nearly everyone else ignores this issue, too... And keep in mind that we try to integrate tech that works at scale - syncthing doesn't look like a thing that works very well with 50.000 users, so that means we wouldn't want to use it for any important feature as most of our users actually can't use it 😺 but maybe I'm wrong about that... |
|
Syncthing scales well. The statistics here show opt-in usage data: https://data.syncthing.net/. The top end of devices reporting show over 100,000 GB being shared in 4k folders over 2,000 devices. The sync protocol in Syncthing is far superior to webdav. Even assuming a server is configured correctly for very large files a webdav upload or download will restart when a client goes offline. Obviously causing issues for laptops and other machines not on 24/7 and making webdav unusable for large files. Syncthing works reliably when resuming extremely large file uploads or downloads. Syncthing also for example can sync huge changed files without needing to reupload the entire file (delta sync). The Syncthing sync protocol is faster than webdav. Better put webdav is not a fast protocol (notwithstanding anything to do with how fast the Syncthing sync protocl is). Both protocols will max out bandwidth eventually during a sync but Syncthing will start utilizing max bandwidth faster than webdav. Syncthing is the closet thing to cross platform real time rsync. Again though Syncthing can be fairly easily integrated into Nextcloud as it is. |
|
I'm not sure these were actually answered, but just to spell it out. LANSync: The syncthing protocol is decentralised, it does LANSync by default. Connections are made directly between synthing instances, there is no central server. If the instances are on the same LAN, they notice this and sync directly over LAN. If they are not on the same LAN they sync over the internet. deltasync: Syncthing syncs file deltas, not whole files. This reduces network traffic and increases speed in comparison to webdav. Syncthing is being used to sync crazy amounts of data already (see the stats link from schnappijedi above), and has been doing so for years. I've used it since Syncthing version 10 and it has been extremely reliable the whole time. The main problem with Syncthing is that it is not multi-user (each user on a system must run their own instance of Syncthing to sync their own files), so I'm not sure how Syncthing would work for Nextcloud servers with multiple users. I don't really see how it can be used for multiple user clouds when running as www-data. I suspect integrating syncthing with Nextcloud would require extracting the protocol (maybe there's a syncthing library that can be used) and using that directly within the nextcloud server and client. The other problem with Syncthing is that they chose to implement it in the Go programming language. Those things sound like a lot of work, but on the other hand, if you thought the features were worth it, it could save a lot of work in creating a robust version of the same thing yourselves. |
|
@crobarcro great point. Syncthing is device level security model while Nextcloud is user level security model. Manual Syncthing integration will work great as long as it is understood that anyone with access to Syncthing also has access to any and all files owned by www-data (or user running the Nextcloud web server). |
|
The scalability answer isn't very convincing - sure, syncthing works with 2000 devices. Enough for, say, 1000 users. That is a medium size customer of ours - bigger ones have 100.000 users, say 150.000 devices. Some customers have millions of users. Then about using the syncthing protocol in our client instead of webDAV. We picked webDAV because it is a standard that is widely supported and works through any firewall. Syncthing uses its own, special protocol - which you can't just mount in mac or windows, nor use from third party apps, and if it doesn't use port 443 it won't work through many firewalls. Also, syncthing isn't a PHP app so it can't be installed easily on Nextcloud or integrated, especially in a large scale environment. Sorry, it seems a nice non-enterprise, home user tech. Not really suitable for Nextcloud which has to work in a serious setup. Look, if it is delta-sync and lan-sync you want, those are perfectly possible in Nextcloud. Just not done. Find a C++ developer and bug him/her to contribute these features to Nextcloud. Heck, deltasync is already mostly done and its developer wants to put it in Nextcloud - LANSync can be done, too. |
|
Agree with decision not to integrate as the two projects do not officially fit together for a great many reasons. This being said webdav is a/the protocol that does not scale well (large number of large files, really anything over 100 MB or so, is a no go with webdav) and is the reason many choose to integrate Syncthing manually. Syncthing is harder to setup, but really isn't different from any other protocol, just as webdav, in that it is dependent on running on port 443 to punch through many firewalls. Syncthing solves this through relay servers turned on by default but that isn't relevant here. Syncthing also isn't built in to any operating system but the bottom line is that one will not experience issues with the actual sync process, regardless of scale, size or number of files, or on a LAN or not, once Syncthing is setup. Regular people cannot setup Syncthing (not because they can't but because it isn't the easiest, better put the entire Syncthing install/ setup isn't what people are used to which inherently makes it more difficult) but once setup Syncthing scales and works far better than webdav. Referenced opt-in usage statistics to support scalability of Syncthing. There are many articles and usage cases attesting to the scalability of webdav. Would encourage all interested run both webdav and Syncthing to draw their own conclusions. Still cannot really blame the choice of webdav in Nextcloud. Anyone would be hard pressed to figure out a better easy alternative. But webdav is certainly very far from the best syncing option for anyone who wants a better final product in exchange for going through a more difficult and complicated setup. |
|
My experience is that Syncthing is vastly easier to install as it only took me about 30 seconds + a couple minutes to configure. My thought is it would be great to allow these open protocols to more seamlessly integrate. I want to point out that Syncthing also scales down really well for smaller user cases or in the event a lamp stack goes down. Home users should be seriously important to Nextcloud. Device to device p2p sync is useful. Nextcloud documentation/official administrative tools are less detailed / relevant for those of us with few users, and I find both projects offer different benefits to each other. Thanks for reading and all the great work. |
|
@sunjam Maybe you are coming at this the wrong way. Syncthing was never going to be integrated into the Nextcloud client. Nextcloud has paying corporate clients and salaried/ contract employees working on the webdav client implementation in addition to years of work being made irrelevant if this (major) change took place. Basically Nextcloud is invested in webdav and cannot pivot like this even if they wanted to. Why don't you approach this by looking at Nextcloud as complimentary to a Syncthing node instead of the other way around? It is conceivable Syncthing could implement simple file access through the web GUI in the future. Maybe this request could be made to Syncthing who may be more open and agile to change. This would leave only the sharing of files via links and multi user access to a single node as to functionality that Nextcloud would bring to Syncthing. Especially in a small environment these things may not be necessary thereby allowing the utilization of Syncthing alone without Nextcloud. Secondly maybe Syncthing can independently publish an app to Nextcloud (https://apps.nextcloud.com/) adding the functionality of Syncthing to Nextcloud. Misspoke slightly in terms of "easy." Notwithstanding the server aspect of both Syncthing and Nextcloud and assuming one server/node of each already exists the Nextcloud client is "easier" to setup than a new Syncthing node. Nextcloud uses a standard username and password which people are used to. Going to locahost:8384 only to be further confused when Google search results are shown because http:// wasn't used or the search bar was used instead of the address bar is difficult the first time one experiences it. A lot of people might be done at this point and dissatisfied. It is solely for this reason that say the Nextcloud client setup is "easier" than a Syncthing node. If all apps/ programs used web GUI's the equation would flip. Web GUI's are better, way less bloated, and actually easier since most people are always in a web browser anyways but would venture to say that what is "normal" is going to be preferred by most. |
|
@schnappijedi I’m just thinking of the community aspect of Nextcloud as opposed to the corporate and don’t mean to suggest an end-all pivot away from webdav. An independent app would be amazing. My use is as a “non-enterprise, home user” and I feel there are many such users that are serious about the Nextcloud community. I totally understand that enterprise leads the charge; I just read some level of being dismissed as a member of the general Nextcloud community. Still love the software + am pumped for the future. |
|
With you wholeheartedly on the community aspect. Another way in which Nextcloud could learn and borrow from Syncthing. |
|
I suggest to open another issue to keep track of the LAN sync feature request in general (this one was closed because of the syncthing implementation detail). |
|
@zarelit LAN Sync discussion is currently taking place in this Owncloud thread. |
|
@sunjam Sorry I assumed one issue per project was needed (like ∆-sync). I'll follow the OC one, thank you! |
|
Lots of wrong information, but one thing I know is correct: I will go the rest of my life and not see a post as ignorant and short sighted as @jospoortvliet 's post |
|
Duh, yes myopic! @jospoortvliet misses the point completely. The original request is asking how to use syncthing to share files across 1000 servers, not 1000 users. Sure some organisations have 100000+ users, CERN, but they can achieve this with less than 10 servers and a centralised ceph based filesystem. Combining syncthing and nextcloud was suggested as a way to support millions of users world wide, server to server (not client to server) |
Would provide a solution for issue #417 requesting longterm delta sync and issue #230 for Nextcloud and Owncloud requesting LAN Sync. Also supports transferring more than one file simultaneously. Would be fantastic if these services all played nice together.
Link to discussion in Nextcloud Forum.
Link to discussion in Syncthing Forum.
The text was updated successfully, but these errors were encountered: