Skip to content
This repository has been archived by the owner on Sep 20, 2024. It is now read-only.

Sync Server beginning of documentation #1471

Merged
merged 3 commits into from
May 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions website/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
When developing on Windows make sure `start.sh` has the correct line endings (`LF`).

Start via yarn:
---------------
Clone repository

Install yarn if not already installed (https://classic.yarnpkg.com/en/docs/install)
For example via npm (but could be installed differently too)

```npm install --global yarn```

Then go to `website` folder

```yarn install``` (takes a while)

To start local test server:

```yarn start```

Server is accessible by default on http://localhost:3000

Start via docker:
-----------------
Setting for docker container:
```bash
docker build . -t pype-docs
docker run --rm -p 3000:3000 -v /c/Users/admin/openpype.io:/app pype-docs
```
48 changes: 48 additions & 0 deletions website/docs/artist_tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,22 @@ You can set group of selected subsets with shortcut `Ctrl + G`.
You'll set the group in Avalon database so your changes will take effect for all users.
:::

### Site Sync support

If **Site Sync** is enabled additional widget is shown in right bottom corner.
It contains list of all representations of selected version(s). It also shows availability of representation files
on particular site (*active* - mine, *remote* - theirs).

![site_sync_support](assets/site_sync_loader.png)

On this picture you see that representation files are available only on remote site (could be GDrive or other).
If artist wants to work with the file(s) they need to be downloaded first. That could be done by right mouse click on
particular representation (or multiselect all) and select *Download*.

This will mark representation to be download which will happen in the background if OpenPype Tray is running.

For more details of progress, state or possible error details artist should open **[Sync Queue](#Sync-Queue)** item in Tray app.

Work in progress...

## Library Loader
Expand Down Expand Up @@ -412,3 +428,35 @@ It might also happen that user deletes underlying host item(for example layer in
This could result in phantom issues during publishing. Use Subset Manager to purge workfile from abandoned items.

Please check behaviour in host of your choice.

## Sync Queue

### Details

If **Site Sync** is configured for a project, each asset is marked to be synchronized to a remote site during publishing.
Each artist's OpenPype Tray application handles synchronization in background, it looks for all representation which
are marked with the site of the user (unique site name per artist) and remote site.

Artists then can see progress of synchronization via **Sync Queue** link in the Tray application.

Artists can see all synced representation in this dialog with helpful information such as when representation was created, when it was synched,
status of synchronization (OK or Fail) etc.

### Usage

With this app artists can modify synchronized representation, for example mark failed representation for re-sync etc.

![Sync Queue](assets/site_sync_sync_queue.png)

Actions accessible by context menu on single (or multiple representations):
- *Open in Explorer* - if site is locally accessible, open folder with it with OS based explorer
- *Re-sync Active Site* - mark artist own side for re-download (repre must be accessible on remote side)
- *Re-sync Remote Site* - mark representation for re-upload
- *Completely remove from local* - removes tag of synchronization to artist's local site, removes files from disk (available only for personal sites)
- *Change priority* - mark representations with higher priority for faster synchronization run

Double click on any of the representation open Detail dialog with information about all files for particular representation.
In this dialog error details could be accessed in the context menu.

Artists can also Pause whole server or specific project for synchronization. In that state no download/upload is being run.
This might be helpful if the artist is not interested in a particular project for a while or wants to save bandwidth data limit for a bit.
Binary file added website/docs/assets/site_sync_gdrive_user.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/docs/assets/site_sync_loader.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/docs/assets/site_sync_local_setting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/docs/assets/site_sync_sync_queue.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/docs/assets/site_sync_system.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
102 changes: 102 additions & 0 deletions website/docs/module_site_sync.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,105 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';


:::warning
**This feature is** currently **in a beta stage** and it is not recommended to rely on it fully for production.
:::

Site Sync allows users and studios to synchronize published assets between multiple 'sites'. Site denotes a storage location,
which could be a physical disk, server, cloud storage. To be able to use site sync, it first needs to be configured.

The general idea is that each user acts as an individual site and can download and upload any published project files when they are needed. that way, artist can have access to the whole project, but only every store files that are relevant to them on their home workstation.

:::note
At the moment site sync is only able to deal with publishes files. No workfiles will be synchronized unless they are published. We are working on making workfile synchronization possible as well.
:::

## System Settings

To use synchronization, *Site Sync* needs to be enabled globally in **OpenPype Settings/System/Modules/Site Sync**.

![Configure module](assets/site_sync_system.png)


## Project Settings

Sites need to be made available for each project. Of course this is possible to do on the default project as well, in which case all other projects will inherit these settings until overriden explicitly.

You'll find the setting in **Settings/Project/Global/Site Sync**

The attributes that can be configured will vary between sites and their providers.

## Local settings

Each user should configure root folder for their 'local' site via **Local Settings** in OpenPype Tray. This folder will be used for all files that the user publishes or downloads while working on a project. Artist has the option to set the folder as "default"in which case it is used for all the projects, or it can be set on a project level individually.

Artists can also override which site they use as active and remote if need be.

![Local overrides](assets/site_sync_local_setting.png)


## Sites

By default there are two sites created for each OpenPype installation:
- **studio** - default site - usually a centralized mounted disk accessible to all artists. Studio site is used if Site Sync is disabled.
- **local** - each workstation or server running OpenPype Tray receives its own with unique site name. Workstation refers to itself as "local"however all other sites will see it under it's unique ID.

Artists can explore their site ID by opening OpenPype Info tool by clicking on a version number in the tray app.

Many different sites can be created and configured on the system level, and some or all can be assigned to each project.

Each OpenPype Tray app works with two sites at one time. (Sites can be the same, and no synching is done in this setup).

Sites could be configured differently per project basis.


## Providers

Each site implements a so called `provider` which handles most common operations (list files, copy files etc.) and provides interface with a particular type of storage. (disk, gdrive, aws, etc.)
Multiple configured sites could share the same provider with different settings (multiple mounted disks - each disk can be a separate site, while
all share the same provider).

**Currently implemented providers:**

### Local Drive

Handles files stored on disk storage.

Local drive provider is the most basic one that is used for accessing all standard hard disk storage scenarios. It will work with any storage that can be mounted on your system in a standard way. This could correspond to a physical external hard drive, network mounted storage, internal drive or even VPN connected network drive. It doesn't care about how te drive is mounted, but you must be able to point to it with a simple directory path.

Default sites `local` and `studio` both use local drive provider.


### Google Drive

Handles files on Google Drive (this). GDrive is provided as a production example for implementing other cloud providers

Let's imagine a small globally distributed studio which wants all published work for all their freelancers uploaded to Google Drive folder.

For this use case admin needs to configure:
- how many times it tries to synchronize file in case of some issue (network, permissions)
- how often should synchronization check for new assets
- sites for synchronization - 'local' and 'gdrive' (this can be overriden in local settings)
- user credentials
- root folder location on Google Drive side

Configuration would look like this:

![Configure project](assets/site_sync_project_settings.png)

*Site Sync* for Google Drive works using its API: https://developers.google.com/drive/api/v3/about-sdk

To configure Google Drive side you would need to have access to Google Cloud Platform project: https://console.cloud.google.com/

To get working connection to Google Drive there are some necessary steps:
- first you need to enable GDrive API: https://developers.google.com/drive/api/v3/enable-drive-api
- next you need to create user, choose **Service Account** (for basic configuration no roles for account are necessary)
- add new key for created account and download .json file with credentials
- share destination folder on the Google Drive with created account (directly in GDrive web application)
- add new site back in OpenPype Settings, name as you want, provider needs to be 'gdrive'
- distribute credentials file via shared mounted disk location

### Custom providers

If a studio needs to use other services for cloud storage, or want to implement totally different storage providers, they can do so by writing their own provider plugin. We're working on a developer documentation, however, for now we recommend looking at `abstract_provider.py`and `gdrive.py` inside `openpype/modules/sync_server/providers` and using it as a template.