Update mount volume locations in CVAT

[rushtehrani]

CVAT currently requires multiple volume mounts. We need to reduce these so that more machine types can be supported.

In order to achieve this, we need to:

1. Add environment variables for share, data, logs, keys, models (note that this will be deprecated with APIs)

2. Update CVAT template to only have 3 mounted volumes, one for db, one for share and one for data, logs, keys and models. To support this, we should mount share volume to /share and volume for data, logs, keys and models to /cvat. The environment variables will then be set to /share, /cvat/data, /cvat/logs, /cvat/keys and /cvat/models respectively.

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/enhancement to this issue, with a confidence of 0.84. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[aleksandrmelnikov]

Link to cvat issue: cvat-ai/cvat#2362

[aleksandrmelnikov]

@rushtehrani
Try this template out:

# Workspace arguments
arguments:
  parameters:
    - name: sync-directory
      displayName: Directory to sync raw input and training output
      value: workflow-data
      hint: Location to sync raw input, models and checkpoints from default object storage. Note that this will be relative to the current namespace.
containers:
  - name: cvat-db
    image: postgres:10-alpine
    env:
      - name: POSTGRES_USER
        value: root
      - name: POSTGRES_DB
        value: cvat
      - name: POSTGRES_HOST_AUTH_METHOD
        value: trust
      - name: PGDATA
        value: /var/lib/psql/data
    ports:
      - containerPort: 5432
        name: tcp
    volumeMounts:
      - name: db
        mountPath: /var/lib/psql
  - name: cvat-redis
    image: redis:4.0-alpine
    ports:
      - containerPort: 6379
        name: tcp
  - name: cvat
    image: onepanel/cvat:0.14.0_cvat.alex
    env:
      - name: DJANGO_MODWSGI_EXTRA_ARGS
        value: ""
      - name: ALLOWED_HOSTS
        value: '*'
      - name: CVAT_REDIS_HOST
        value: localhost
      - name: CVAT_POSTGRES_HOST
        value: localhost
      - name: CVAT_SHARE_URL
        value: /cvat/data
      - name: CVAT_SHARE_DIR
        value: /share
      - name: CVAT_KEYS_DIR
        value: /cvat/keys
      - name: CVAT_DATA_DIR
        value: /cvat/data
      - name: CVAT_MODELS_DIR
        value: /cvat/models
      - name: CVAT_LOGS_DIR
        value: /cvat/logs
      - name: ONEPANEL_SYNC_DIRECTORY
        value: '{{workspace.parameters.sync-directory}}'
      - name: NVIDIA_VISIBLE_DEVICES
        value: all
      - name: NVIDIA_DRIVER_CAPABILITIES
        value: compute,utility
      - name: NVIDIA_REQUIRE_CUDA
        value: "cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411"
    ports:
      - containerPort: 8080
        name: http
    volumeMounts:
      - name: data-logs-keys-models
        mountPath: /cvat
      - name: share
        mountPath: /share
      - name: sys-namespace-config
        mountPath: /etc/onepanel
        readOnly: true
  - name: cvat-ui
    image: onepanel/cvat-ui:0.14.0_cvat.1.0.0
    ports:
      - containerPort: 80
        name: http
  # You can add multiple FileSyncer sidecar containers if needed
  - name: filesyncer
    image: onepanel/filesyncer:s3
    imagePullPolicy: Always
    args:
      - download
      - -server-prefix=/sys/filesyncer
    env:
      - name: FS_PATH
        value: /mnt/share
      - name: FS_PREFIX
        value: '{{workflow.namespace}}/{{workspace.parameters.sync-directory}}'
    volumeMounts:
      - name: share
        mountPath: /mnt/share
      - name: sys-namespace-config
        mountPath: /etc/onepanel
        readOnly: true
ports:
  - name: cvat-ui
    port: 80
    protocol: TCP
    targetPort: 80
  - name: cvat
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: fs
    port: 8888
    protocol: TCP
    targetPort: 8888
routes:
  - match:
      - uri:
          prefix: /sys/filesyncer
    route:
      - destination:
          port:
            number: 8888
  - match:
      - uri:
          regex: /api/.*|/git/.*|/tensorflow/.*|/onepanelio/.*|/tracking/.*|/auto_annotation/.*|/analytics/.*|/static/.*|/admin/.*|/documentation/.*|/dextr/.*|/reid/.*
      - queryParams:
          id:
            regex: \d+.*
    route:
      - destination:
          port:
            number: 8080
  - match:
      - uri:
          prefix: /
    route:
      - destination:
          port:
            number: 80
# DAG Workflow to be executed once a Workspace action completes (optional)
# Uncomment the lines below if you want to send Slack notifications
#postExecutionWorkflow:
#  entrypoint: main
#  templates:
#  - name: main
#    dag:
#       tasks:
#       - name: slack-notify
#         template: slack-notify
#  - name: slack-notify
#     container:
#       image: technosophos/slack-notify
#       args:
#       - SLACK_USERNAME=onepanel SLACK_TITLE="Your workspace is ready" SLACK_ICON=https://www.gravatar.com/avatar/5c4478592fe00878f62f0027be59c1bd SLACK_MESSAGE="Your workspace is now running" ./slack-notify
#       command:
#       - sh
#       - -c

[rushtehrani]

@aleksandrmelnikov everything seems to work except for some reason file progress message from filesyncer is not showing up anymore. cc @Vafilor

Also, here's the slightly updated template I'd like to use:

# Workspace arguments
arguments:
  parameters:
    - name: sync-directory
      displayName: Directory to sync raw input and training output
      value: workflow-data
      hint: Location (relative to current naemspace) to sync raw input, models and checkpoints from default object storage to '/share'.
containers:
  - name: cvat-db
    image: postgres:10-alpine
    env:
      - name: POSTGRES_USER
        value: root
      - name: POSTGRES_DB
        value: cvat
      - name: POSTGRES_HOST_AUTH_METHOD
        value: trust
      - name: PGDATA
        value: /var/lib/psql/data
    ports:
      - containerPort: 5432
        name: tcp
    volumeMounts:
      - name: db
        mountPath: /var/lib/psql
  - name: cvat-redis
    image: redis:4.0-alpine
    ports:
      - containerPort: 6379
        name: tcp
  - name: cvat
    image: onepanel/cvat:0.14.0_cvat.alex
    env:
      - name: DJANGO_MODWSGI_EXTRA_ARGS
        value: ""
      - name: ALLOWED_HOSTS
        value: '*'
      - name: CVAT_REDIS_HOST
        value: localhost
      - name: CVAT_POSTGRES_HOST
        value: localhost
      - name: CVAT_SHARE_URL
        value: /cvat/data
      - name: CVAT_SHARE_DIR
        value: /share
      - name: CVAT_KEYS_DIR
        value: /cvat/keys
      - name: CVAT_DATA_DIR
        value: /cvat/data
      - name: CVAT_MODELS_DIR
        value: /cvat/models
      - name: CVAT_LOGS_DIR
        value: /cvat/logs
      - name: ONEPANEL_SYNC_DIRECTORY
        value: '{{workspace.parameters.sync-directory}}'
      - name: NVIDIA_VISIBLE_DEVICES
        value: all
      - name: NVIDIA_DRIVER_CAPABILITIES
        value: compute,utility
      - name: NVIDIA_REQUIRE_CUDA
        value: "cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411"
    ports:
      - containerPort: 8080
        name: http
    volumeMounts:
      - name: cvat-data
        mountPath: /cvat
      - name: share
        mountPath: /share
      - name: sys-namespace-config
        mountPath: /etc/onepanel
        readOnly: true
  - name: cvat-ui
    image: onepanel/cvat-ui:0.14.0_cvat.1.0.0
    ports:
      - containerPort: 80
        name: http
  # You can add multiple FileSyncer sidecar containers if needed
  - name: filesyncer
    image: onepanel/filesyncer:s3
    imagePullPolicy: Always
    args:
      - download
      - -server-prefix=/sys/filesyncer
    env:
      - name: FS_PATH
        value: /mnt/share
      - name: FS_PREFIX
        value: '{{workflow.namespace}}/{{workspace.parameters.sync-directory}}'
    volumeMounts:
      - name: share
        mountPath: /mnt/share
      - name: sys-namespace-config
        mountPath: /etc/onepanel
        readOnly: true
ports:
  - name: cvat-ui
    port: 80
    protocol: TCP
    targetPort: 80
  - name: cvat
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: fs
    port: 8888
    protocol: TCP
    targetPort: 8888
routes:
  - match:
      - uri:
          prefix: /sys/filesyncer
    route:
      - destination:
          port:
            number: 8888
  - match:
      - uri:
          regex: /api/.*|/git/.*|/tensorflow/.*|/onepanelio/.*|/tracking/.*|/auto_annotation/.*|/analytics/.*|/static/.*|/admin/.*|/documentation/.*|/dextr/.*|/reid/.*
      - queryParams:
          id:
            regex: \d+.*
    route:
      - destination:
          port:
            number: 8080
  - match:
      - uri:
          prefix: /
    route:
      - destination:
          port:
            number: 80
volumeClaimTemplates:
- metadata:
    name: db
  spec:
    accessModes: ["ReadWriteOnce"]
    resources:
      requests:
        storage: 20Gi
# DAG Workflow to be executed once a Workspace action completes (optional)
# Uncomment the lines below if you want to send Slack notifications
#postExecutionWorkflow:
#  entrypoint: main
#  templates:
#  - name: main
#    dag:
#       tasks:
#       - name: slack-notify
#         template: slack-notify
#  - name: slack-notify
#     container:
#       image: technosophos/slack-notify
#       args:
#       - SLACK_USERNAME=onepanel SLACK_TITLE="Your workspace is ready" SLACK_ICON=https://www.gravatar.com/avatar/5c4478592fe00878f62f0027be59c1bd SLACK_MESSAGE="Your workspace is now running" ./slack-notify
#       command:
#       - sh
#       - -c

[Vafilor]

@rushtehrani I'll take a look

[rushtehrani]

@aleksandrmelnikov we still need a migration for this to be completed. Also note my updated template.