From 7f9a4e594282d6d7e842cd259a5755441f5d3ebd Mon Sep 17 00:00:00 2001 From: Carla Schroder Date: Tue, 22 Mar 2016 17:28:51 -0700 Subject: [PATCH] update occ commands for external storage --- .../external_storage_configuration.rst | 525 +----------------- .../configuration_server/occ_command.rst | 16 +- 2 files changed, 15 insertions(+), 526 deletions(-) diff --git a/admin_manual/configuration_files/external_storage_configuration.rst b/admin_manual/configuration_files/external_storage_configuration.rst index ad066a2dd..3f3c727bc 100644 --- a/admin_manual/configuration_files/external_storage_configuration.rst +++ b/admin_manual/configuration_files/external_storage_configuration.rst @@ -2,525 +2,6 @@ Configuring External Storage (Configuration File) ================================================= -You may configure external storage mounts by creating and editing the -:file:`data/mount.json`. This file contains all settings in JSON (JavaScript -Object Notation) format. At the moment two different types of entries exist: - -- **Group mounts:** each entry configures a mount for each user in group. -- **User mounts:** each entry configures a mount for a single user or for all - users. - -For each type, there is a JSON array with the user/group name as key, and an -array of configuration entries as value. Each entry consists of the class name -of the storage backend and an array of backend specific options and will be -replaced by the user login. The template **$user** can be used in the mount -point or backend options. As of writing the following storage backends are -available for use: - -- Local file system -- FTP (or FTPS) -- SFTP -- SMB -- WebDAV -- `Amazon S3`_ -- `Dropbox`_ -- `Google Drive`_ -- `OpenStack Swift`_ - -You need to enable the `External storage support` app first. You can -do this on the Apps page of your ownCloud Web GUI, or use the ``occ`` command. -This example shows how to list apps, and how to enable the `External storage -support` app on Debian Linux and its derivatives:: - - $ sudo -u www-data php occ app:list - Enabled: - [snip] - Disabled: - - files_external - [snip] - - $ sudo -u www-data php occ app:enable files_external - files_external enabled - -See :doc:`../configuration_server/occ_command` to learn more about ``occ``. - -.. note:: A non-blocking or correctly configured SELinux setup is needed - for these backends to work. Please refer to the :ref:`selinux-config-label`. - -It is recommended to use the :doc:`Web-GUI ` in the -administrator panel to add, remove or modify mount options to prevent any problems! - -Using self-signed certificates ------------------------------- - -When using self-signed certificates for external storage mounts the certificate -needs to be imported in the personal settings of the user. Please refer to `this -`_ -blogpost for more information. - -Adding files to external storages ---------------------------------- - -In general it is recommended to configure the background job ``Webcron`` or -``Cron`` as described in :doc:`../configuration_server/background_jobs_configuration` -so ownCloud is able to detect files added to your external storages without the need -for a user to be browsing your ownCloud installation. - -Please also be aware that ownCloud might not always be able to find out what has been -changed remotely (files changed without going through ownCloud), especially -when it's very deep in the folder hierarchy of the external storage. - -You might need to setup a cron job that runs ``sudo -u www-data php occ -files:scan --all`` (or replace "--all" with the user name, see also -:doc:`../configuration_server/occ_command`) to trigger a rescan of the user's -files periodically (for example every 15 minutes), which includes the mounted -external storage. - -Example -------- - -:: - - { - "group":{ - "admin":{ - "\/$user\/files\/Admin_Stuff":{ - "class":"\\OC\\Files\\Storage\\Local", - "options":{ ... }, - "priority":150 - } - } - }, - "user":{ - "all":{ - "\/$user\/files\/Pictures":{ - "class":"\\OC\\Files\\Storage\\DAV", - "options":{ ... }, - "priority":100 - } - }, - "someuser":{ - "\/someuser\/files\/Music":{ - "class":"\\OC\\Files\\Storage\\FTP", - "options":{ ... }, - "priority":100 - } - } - } - } - -Priorities ----------- - -An advanced feature is available, only configurable directly in -:file:`data/mount.json`, which allows mount configurations to have an associated -priority. When two or more valid mount configurations exist for the same mount point, -the one with the highest priority (defined by the largest number) will take precedence -and become the active mount for the user. - -Each backend has a default priority, assigned when a mount configuration with that -backend is created. The default priority will be shown in the example section for -each backend below. Should a backend not provide a default priority, a value of 100 -will be used. - -There is also a concept of priority types, to preserve compatibility with -previous mount configuration parsing. Mount configurations are evaluated in the -following order, with later mount types always overriding a previous mount type: - -- user -> all : global mount configurations -- group : group mount configurations -- user (not all) : per-user mount configurations -- :file:`data/$user/mount.json` : personal mount configurations - -Configuring Temporary Disk Space Needs --------------------------------------- - -Not all external storage types are currently enabled for, or support -streaming. Therefore ownCloud needs temporary space to buffer data for -transfers. This can occur when there are many concurrent users transferring data -with a higher volume over small bandwidth. ownCloud may need, in these -cases, additional temporary space. - -Example: 100 concurrent users uploading each a 300MB file with a -total transfer time of 6000s (1h 40min). The temporary space needed by ownCloud -for this period of time is 30GB. Even though it is not mandatory, the location -of the temp directory used by ownCloud can be configured manually. To do so, -you need to maintain the ``tempdirectory`` parameter described in -``config.sample.php`` - -As of writing, following external storage list uses temp files for up/download: - -* FTP -* SMB / SMB_OC -* WebDAV -* Amazon S3 -* Dropbox -* Google Drive -* OpenStack SWIFT - -External storage list that uses direct file streaming: - -* Local -* SFTP - -Backends --------- - -Local Filesystem -~~~~~~~~~~~~~~~~ - -The local filesystem backend mounts a folder on the server into the virtual -filesystem, the class to be used is **\\OC\\Files\\Storage\\Local**\ and -takes the following options: - -- **datadir** : the path to the local directory to be mounted - - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\Local", - "options":{ "datadir":"\/mnt\/additional_storage" }, - "priority":150 - } - -.. note:: You must ensure that the Web server has sufficient permissions on the folder. - -FTP (or FTPS) -~~~~~~~~~~~~~ - -The FTP backend mounts a folder on a remote FTP server into the virtual -filesystem and is part of the ‘External storage support’ app, the class -to be used is **\\OC\\Files\\Storage\\FTP**\ and takes the following -options: - -- **host**: the hostname of the ftp server, and optionally the port number -- **user**: the username used to login to the ftp server -- **password**: the password to login to the ftp server -- **secure**: whether to use ftps:// (FTP over TLS) to connect to the ftp - server instead of ftp:// (optional, defaults to false) -- **root**: the remote subfolder inside the ftp server to mount (optional, defaults - to ‘/’) - - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\FTP", - "options":{ - "host":"ftp.myhost.com:21", - "user":"johndoe", - "password":"secret", - "root":"\/Videos", - "secure":"false" - }, - "priority":100 - } - -.. note:: PHP needs to be built with FTP support for this backend to work. - -.. note:: The external storage ``FTP/FTPS/SFTP`` needs the ``allow_url_fopen`` PHP - setting to be set to ``1``. When having connection problems make sure that it is - not set to ``0`` in your ``php.ini``. - -SFTP -~~~~ - -The SFTP backend mounts a folder on a remote SSH server into the virtual -filesystem and is part of the ‘External storage support’ app. The class -to be used is **\\OC\\Files\\Storage\\SFTP**\ and takes the following -options: - -- **host**: the hostname of the SSH server -- **user**: the username used to login to the SSH server -- **password**: the password to login to the SSH server -- **root**: the remote subfolder inside the SSH server to mount (optional, defaults - to ‘/’) - - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\SFTP", - "options":{ - "host":"ssh.myhost.com", - "user":"johndoe", - "password":"secret", - "root":"\/Books" - }, - "priority":100 - } - -.. note:: PHP needs to be built with SFTP support for this backend to work. - -.. note:: The external storage ``FTP/FTPS/SFTP`` needs the ``allow_url_fopen`` PHP - setting to be set to ``1``. When having connection problems make sure that it is - not set to ``0`` in your ``php.ini``. - -SMB -~~~ -The SMB backend mounts a folder on a remote Samba server, a NAS appliance or -a Windows machine into the virtual file system. This requires -``php5-libsmbclient`` (`installation instructions -`_). -It is part of the ‘External storage support’ app, the class to be used -is **\\OC\\Files\\Storage\\SMB**\ and takes the following options: - -- **host**: the host name of the samba server -- **user**: the username or domain/username to login to the samba server -- **password**: the password to login to the samba server -- **share**: the share on the samba server to mount -- **root**: the remote subfolder inside the samba share to mount (optional, defaults - to ‘/’). To assign the ownCloud logon username automatically to the subfolder, use ``$user`` instead of a particular subfolder name. - -Example -^^^^^^^ -With username only: - -:: - - { "class":"\\OC\\Files\\Storage\\SMB", - "options":{ - "host":"myhost.com", - "user":"johndoe", - "password":"secret", - "share":"\/test", - "root":"\/Pictures" - }, - "priority":100 - } - -With domainname and username: - -:: - - { "class":"\\OC\\Files\\Storage\\SMB", - "options":{ - "host":"myhost.com", - "user":"domain\/johndoe", - "password":"secret", - "share":"\/test", - "root":"\/Pictures" - }, - "priority":100 - } - -WebDAV -~~~~~~ - -The WebDAV backend mounts a folder on a remote WebDAV server into the -virtual filesystem and is part of the ‘External storage support’ app, -the class to be used is **\\OC\\Files\\Storage\\DAV**\ and takes the -following options: - -- **host**: the hostname of the webdav server. -- **user**: the username used to login to the webdav server -- **password**: the password to login to the webdav server -- **secure**: whether to use https:// to connect to the webdav server - instead of http:// (optional, defaults to false) -- **root**: the remote subfolder inside the webdav server to mount (optional, - defaults to ‘/’) - - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\DAV", - "options":{ - "host":"myhost.com\/webdav.php", - "user":"johndoe", - "password":"secret", - "secure":"true" - }, - "priority":100 - } - -Amazon S3 -~~~~~~~~~ - -The Amazon S3 backend mounts a bucket in the Amazon cloud into the virtual -filesystem and is part of the ‘External storage support’ app, the class to -be used is **\\OC\\Files\\Storage\\AmazonS3**\ and takes the following -options: - -- **key**: the key to login to the Amazon cloud -- **secret**: the secret to login to the Amazon cloud -- **bucket**: the bucket in the Amazon cloud to mount - - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\AmazonS3", - "options":{ - "key":"key", - "secret":"secret", - "bucket":"bucket" - }, - "priority":100 - } - -Dropbox -~~~~~~~ - -The Dropbox backend mounts a dropbox in the Dropbox cloud into the virtual -filesystem and is part of the ‘External storage support’ app, the class to -be used is **\\OC\\Files\\Storage\\Dropbox**\ and takes the following options: - -- **configured**: whether the drive has been configured or not (true or false) -- **app_key**: the app key to login to your Dropbox -- **app_secret**: the app secret to login to your Dropbox -- **token**: the OAuth token to login to your Dropbox -- **token_secret**: the OAuth secret to login to your Dropbox - - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\Dropbox", - "options":{ - "configured":"#configured", - "app_key":"key", - "app_secret":"secret", - "token":"#token", - "token_secret":"#token_secret" - }, - "priority":100 - } - -Google Drive -~~~~~~~~~~~~ - -The Google Drive backend mounts a share in the Google cloud into the virtual -filesystem and is part of the ‘External storage support’ app, the class to -be used is **\\OC\\Files\\Storage\\Google**\ and is done via an OAuth2.0 request. -That means that the App must be registered through the Google APIs Console. -The result of the registration process is a set of values (incl. client_id, client_secret). -It takes the following options: - -- **configured**: whether the drive has been configured or not (true or false) -- **client_id**: the client id to login to the Google drive -- **client_secret**: the client secret to login to the Google drive -- **token**: a compound value including access and refresh tokens - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\Google", - "options":{ - "configured":"#configured", - "client_id":"#client_id", - "client_secret":"#client_secret", - "token":"#token" - }, - "priority":100 - } - -OpenStack Swift -~~~~~~~~~~~~~~~ - -The Swift backend mounts a container on an OpenStack Object Storage server -into the virtual filesystem and is part of the ‘External storage support’ -app, the class to be used is **\\OC\\Files\\Storage\\SWIFT**\ and takes -the following options: - -- **host**: the hostname of the authentication server for the swift - storage. -- **user**: the username used to login to the swift server -- **token**: the authentication token to login to the swift server -- **secure**: whether to use ftps:// to connect to the swift server instead - of ftp:// (optional, defaults to false) -- **root**: the container inside the swift server to mount (optional, - defaults to ‘/’) - -Example -^^^^^^^ - -:: - - { "class":"\\OC\\Files\\Storage\\SWIFT", - "options":{ - "host":"swift.myhost.com\/auth", - "user":"johndoe", - "token":"secret", - "root":"\/Videos", - "secure":"true" - }, - "priority":100 - } - -External Storage Password Management ------------------------------------- - -ownCloud handles passwords for external mounts differently than regular -ownCloud user passwords. - -The regular user and file share passwords (when you use the default ownCloud -user backend) are stored using a strong cryptographically secure hashing -mechanism in the database. On a new user account with a new password, the -password is hashed and stored in the ownCloud database. The plain-text password -is never stored. When the user logs in, the hash of the password they enter is -compared with the hash in the database. When the hashes match the user is -allowed access. These are not recoverable, so when a user loses a password the -only option is to create a new password. - -Passwords which are used to connect against external storage (e.g. -SMB or FTP), there we have to differentiate again between different -implementations: - -1. **Login with ownCloud credentials** - -When a mountpoint has this option, for example ``SMB / CIFS using OC login``, -the password will be intercepted when a user logs in and written to the PHP -session (which is a file on the filesystem), and written encrypted into the -session with a key from the configuration file. Every time that password is -required ownCloud reads it from the PHP session file. - -When you use this option, features such as sharing will not work properly from -that mountpoint when the user is not logged-in. - -Depending on the implementation of the application, this means that the password -could get leaked in the ``ps`` output, as we use ``smbclient`` for SMB storage -access in the community version. There is a `bug report on this -`_. Consequently, we're currently -evaluating an alternative approach accessing the library directly, and thus not -leaking the password anymore. This is already implemented in the Enterprise -Edition in our Windows Network Drive application, and it will get into the -community version once we have streamlined the code of the ``files_external`` -application a little bit more. - -2. **Stored credentials** - -When you enter credentials into the ``files_external`` dialog those are stored -on the filesystem and encrypted with a key stored in ``config.php``. This is -required since ownCloud needs access to those files and shares even when the -user is not logged-in to have sharing and other key features properly working. - -To sum up: - -The "login with ownCloud credentials" SMB function in the community edition -exposes the password in the server system's process list. If you want to get -around this limitation without waiting for it to be addressed in CE you can get -the Enterprise Edition. However, even then the password is stored in the PHP -session and a malicious admin could access it. You can protect your PHP session -files using protections available in your filesystem. Stored credentials are -always accessible to the ownCloud instance. - -.. _Amazon S3: http://aws.amazon.com/de/s3/ -.. _Dropbox: https://www.dropbox.com/ -.. _Google Drive: https://drive.google.com/start -.. _OpenStack Swift: http://openstack.org/projects/storage/ +Starting with ownCloud 9.0, the :file:`data/mount.json` file for configuring +external storages has been removed, and replaced with a set of +:ref:`occ commands `. diff --git a/admin_manual/configuration_server/occ_command.rst b/admin_manual/configuration_server/occ_command.rst index 5136c0d4e..93d306395 100644 --- a/admin_manual/configuration_server/occ_command.rst +++ b/admin_manual/configuration_server/occ_command.rst @@ -637,6 +637,9 @@ before removing a user:: Files External -------------- +These commands replace the ``data/mount.json`` configuration file used in +ownCloud releases before 9.0. + .. note:: These commands are only available when the "External storage support" app (``files_external``) is enabled. @@ -656,11 +659,16 @@ Commands for managing external storage:: files_external:verify Verify mount configuration These commands replicate the functionality in the ownCloud Web GUI, plus two new -features: ``files_external:export`` and ``files_external:import``. Use -``files_external:export`` to export all admin mounts to a text file, and +features: ``files_external:export`` and ``files_external:import``. + +Use ``files_external:export`` to export all admin mounts to stdout, and ``files_external:export [user_id]`` to export the mounts of the specified -ownCloud user. Use ``files_external:import [filename]`` to import the -configuration. +ownCloud user. + +Use ``files_external:import [filename]`` to import legacy JSON configurations, +and to copy external mount configurations to another ownCloud server. + +Added in 9.0. .. _integrity_check_label: