apps sc: harbor rclone fixes for azure

Added new config option for custom sync and restore to set path
Added new validation to prevent default bucket sync on azure
Added new restore part for harbor on azure

config/sc-config.yaml

@@ -54,17 +54,21 @@ objectStorage:
     ## Restore targets
     targets: []
       # - destinationName: <bucket-or-container-name>
-      #   # destinationType: <object-storage-type> # optional - defaults to main object storage type
-      #   # sourceName: <bucket-or-container-name> # optional - defaults to destination name
-      #   # sourceType: <object-storage-type> # optional - defaults to sync object storage type
+      #   # destinationType: <object-storage-type> # Optional: if set supported values are 's3', 'azure' or 'swift'. Defaults to value of 'objectStorage.type'
+      #   # destinationPath: /folder/name # Optional: Only sync items from this path. Defaults to ""
+      #   # sourceName: <bucket-or-container-name> # Defaults to value of 'destinationName'
+      #   # sourceType: <object-storage-type> # Optional: if set supported values are 's3', 'azure' or 'swift'. Defaults to value of 'objectStorage.sync.destinationType'
+      #   # sourcePath: /folder/name # Optional: Only sync items from this path. Defaults to ""
+      #   # nameSuffix: custom-name # Optional: Used to add a special suffix to the cronjob name. Defaults to "custom"
 
   ## Off-site backup replication between two providers or regions using rclone sync
   sync:
     activeDeadlineSeconds: 14400
     enabled: false
     dryrun: false
 
-    ## If Harbor or Thanos are using Swift then we will automatically use Swift for the sync of Harbor or Thanos, regardless of the value set for destinationType.
+    ## If Harbor or Thanos are using Swift then we will automatically use Swift for the sync of Harbor or Thanos with 'syncDefaultBuckets', regardless of the value set for destinationType.
+    ## Supported values are 's3', 'azure', 'swift'
     destinationType: s3
     # secondaryUrl: set-me if regionEndpoint and or authUrl does not have all the relevant ips and or ports used for rclone-sync networkpolicy.
     # s3:
@@ -87,11 +91,15 @@ objectStorage:
 
     ## Buckets to sync.
     buckets: []
-      # - source: source-bucket
-      #   # destination: destination-bucket # if unset it will be the same as 'source'
-      #   # schedule: 0 5 * * *             # if unset it will be the same as 'defaultSchedule'
-      #   # sourceType: Optional but if included it uses s3 or swift
-      #   # destinationType: Optional but if included it uses s3 or swift.
+      # - source: <bucket-or-container-name>
+      #   # destination: <bucket-or-container-name> # Defaults to value of 'source'
+      #   # schedule: 0 5 * * * # Defaults to value of 'objectStorage.sync.defaultSchedule'
+      #   # sourceType: <object-storage-type> # Optional: if set supported values are 's3', 'azure' or 'swift'. Defaults to value of 'objectStorage.type'
+      #   # sourcePath: /folder/name # Optional: Only sync items from this path. Defaults to ""
+      #   # destinationType: <object-storage-type> # Optional: if set supported values are 's3', 'azure' or 'swift'. Defaults to value of 'objectStorage.sync.destinationType'
+      #   # destinationPath: /folder/name # Optional: Only sync items from this path. Defaults to ""
+      #   # nameSuffix: custom-name # Optional: Used to add a special suffix to the cronjob name. Defaults to "custom"
+
     # Encrypt (symmetric) before syncing
     encrypt:
       enabled: false

config/schemas/config.yaml

@@ -1143,6 +1143,44 @@ allOf:
                   ips:
                     title: Network Policies Ingress Override IPs
                     $ref: '#/$defs/iplist'
+- if:
+    allOf:
+      - properties:
+          harbor:
+            properties:
+              persistence:
+                properties:
+                  type:
+                    const: objectStorage
+          objectStorage:
+            properties:
+              type:
+                const: azure
+      - anyOf:
+          - properties:
+              objectStorage:
+                properties:
+                  sync:
+                    properties:
+                      enabled:
+                        const: true
+          - properties:
+              objectStorage:
+                properties:
+                  restore:
+                    properties:
+                      enabled:
+                        const: true
+                      addTargetsFromSync:
+                        const: true
+  then:
+    properties:
+      objectStorage:
+        properties:
+          sync:
+            properties:
+              syncDefaultBuckets:
+                const: false
 properties:
   global:
     title: Global options
@@ -2339,6 +2377,19 @@ properties:
               description: |-
                 List of buckets to sync when `syncDefaultBuckets` is false
               type: object
+              examples:
+              - source: azure-environment-harbor
+                sourceType: azure
+                sourcePath: //docker
+                destinationPath: /docker
+                destinationType: s3
+                nameSuffix: docker
+              - source: azure-environemnt-harbor
+                sourceType: azure
+                sourcePath: /backups
+                destinationPath: /backups
+                destinationType: s3
+                nameSuffix: backups
               required:
                 - source
               properties:
@@ -2362,13 +2413,33 @@ properties:
                     - s3
                     - swift
                     - azure
+                sourcePath:
+                  type: string
+                  title: Path to sync from
+                  description: |-
+                    Rclone will sync all files from this path.
+                    Defaults to "" (root of bucket).
+                  default: ""
                 destinationType:
                   type: string
                   title: Type of destination
                   examples:
                     - s3
                     - swift
                     - azure
+                destinationPath:
+                  type: string
+                  title: Path to sync to
+                  description: |-
+                    Rclone will sync files to this patch.
+                    Defaults to "" (root of bucket).
+                  default: ""
+                nameSuffix:
+                  type: string
+                  title: Suffix to cronjob name
+                  description: |-
+                    Suffix added to the end of cronjob name.
+                    Defaults to "custom". The default buckets have "default" as suffix
               additionalProperties: false
           destinationType:
             title: Rclone Sync Destination Type
@@ -2472,6 +2543,13 @@ properties:
                     - s3
                     - swift
                     - azure
+                destinationPath:
+                  type: string
+                  title: Path to restore to
+                  description: |-
+                    Rclone will sync files to this patch.
+                    Defaults to "" (root of bucket).
+                  default: ""
                 sourceName:
                   type: string
                   title: Source bucket to sync
@@ -2482,6 +2560,19 @@ properties:
                     - s3
                     - swift
                     - azure
+                sourcePath:
+                  type: string
+                  title: Path to sync from
+                  description: |-
+                    Rclone will sync all files from this path.
+                    Defaults to "" (root of bucket).
+                  default: ""
+                nameSuffix:
+                  type: string
+                  title: Suffix to cronjob name
+                  description: |-
+                    Suffix added to the end of cronjob name.
+                    Defaults to "custom".
               additionalProperties: false
           timestamp:
             title: Rclone Restore Timestamp

helmfile.d/charts/rclone/templates/cronjobs.yaml

@@ -5,7 +5,7 @@ apiVersion: batch/v1
 kind: CronJob
 metadata:
   labels: {{- include "rclone.labels" $ | nindent 4 }}
-  name: {{ printf "%s-%s" (include "rclone.fullname" $) .destinationName | trunc 63 | trimSuffix "-" }}
+  name: {{ printf "%s-%s-%s" (include "rclone.fullname" $) .destinationName .nameSuffix | trunc 63 | trimSuffix "-" }}
 spec:
   concurrencyPolicy: Forbid
   failedJobsHistoryLimit: {{ $.Values.failedJobsHistoryLimit }}
@@ -38,14 +38,14 @@ spec:
               args:
                 - sync
                 {{- if get . "sourceCrypt" | default false }}
-                - "decrypt-{{ .sourceName }}:"
+                - "decrypt-{{ .sourceName }}:{{ .sourcePath }}"
                 {{- else }}
-                - "source-{{ .sourceType }}:{{ .sourceName }}"
+                - "source-{{ .sourceType }}:{{ .sourceName }}{{ .sourcePath }}"
                 {{- end }}
                 {{- if get . "destinationCrypt" | default false }}
-                - "encrypt-{{ .destinationName }}:"
+                - "encrypt-{{ .destinationName }}:{{ .destinationPath }}"
                 {{- else }}
-                - "destination-{{ .destinationType }}:{{ .destinationName }}"
+                - "destination-{{ .destinationType }}:{{ .destinationName }}{{ .destinationPath }}"
                 {{- end }}
                 - --log-level
                 - INFO

helmfile.d/charts/rclone/values.yaml

@@ -37,12 +37,15 @@ crypt:
 # - Providers translate into the configured provider
 targets:
   - destinationName: dn
+    destinationPath: ""
     destinationCrypt: false
     destinationProvider: dp
     sourceName: sn
+    sourcePath: ""
     sourceCrypt: false
     sourceProvider: sp
     schedule: ""
+    nameSuffix: "default"
 
 # Configure rclone providers
 # Prefix destination providers with "destination-" and source providers with "source-"

helmfile.d/values/rclone/restore.yaml.gotmpl

@@ -30,33 +30,42 @@ targets:
 
     {{- $destinations = append $destinations $destinationType }}
     destinationType: {{ $destinationType }}
+    destinationPath: ""
     sourceName: {{ $value }}
     sourceCrypt: {{ $crypt | get "enabled" false }}
     {{- $sources = append $sources $sourceType }}
     sourceType: {{ $sourceType }}
+    sourcePath: ""
+    nameSuffix: default
   {{- end }}
   {{- end }}
 
   {{- range $sync.buckets }}
   - destinationName: {{ .source }}
-    {{- $destinations = get "sourceType" $main.type . | append $destinations }}
-    destinationType: {{ get "sourceType" $main.type . }}
-    sourceName: {{ get "destination" .source . }}
+    {{- $destinations = . | get "sourceType" $main.type | append $destinations }}
+    destinationType: {{ . | get "sourceType" $main.type }}
+    destinationPath: {{ . | get "sourcePath" "" }}
+    sourceName: {{ . | get "destination" .source }}
     sourceCrypt: {{ $crypt | get "enabled" false }}
-    {{- $sources = get "destinationType" $sync.destinationType . | append $sources }}
-    sourceType: {{ get "destinationType" $sync.destinationType . }}
+    {{- $sources = . | get "destinationType" $sync.destinationType | append $sources }}
+    sourceType: {{ . | get "destinationType" $sync.destinationType }}
+    sourcePath: {{ . | get "destinationPath" "" }}
+    nameSuffix: {{ . | get "nameSuffix" "custom" }}
   {{- end }}
 
   {{- end }}
 
   {{- range $restore.targets }}
   - destinationName: {{ .destinationName }}
-    {{- $destinations = get "destinationType" $main.type . | append $destinations }}
-    destinationType: {{ get "destinationType" $main.type . }}
-    sourceName: {{ get "sourceName" .destinationName . }}
+    {{- $destinations = . | get "destinationType" $main.type | append $destinations }}
+    destinationType: {{ . | get "destinationType" $main.type }}
+    destinationPath: {{ . | get "destinationPath" "" }}
+    sourceName: {{ . | get "sourceName" .destinationName }}
     sourceCrypt: {{ $crypt | get "enabled" false }}
-    {{- $sources = get "sourceType" $sync.destinationType . | append $sources }}
-    sourceType: {{ get "sourceType" $sync.destinationType . }}
+    {{- $sources = . | get "sourceType" $sync.destinationType | append $sources }}
+    sourceType: {{ . | get "sourceType" $sync.destinationType }}
+    sourcePath: {{ . | get "sourcePath" "" }}
+    nameSuffix: {{ . | get "nameSuffix" "custom" }}
   {{- end }}
 
 providers:

helmfile.d/values/rclone/sync.yaml.gotmpl

@@ -35,23 +35,29 @@ targets:
 
     {{- $destinations = append $destinations $destinationType }}
     destinationType: {{ $destinationType }}
+    destinationPath: ""
     sourceName: {{ $value }}
     {{- $sources = append $sources $sourceType }}
     sourceType: {{ $sourceType }}
+    sourcePath: ""
+    nameSuffix: default
   {{- end }}
   {{- end }}
 
   {{- range $sync.buckets }}
-  - destinationName: {{ get "destination" .source . }}
+  - destinationName: {{ . | get "destination" .source }}
     destinationCrypt: {{ $crypt | get "enabled" false }}
-    {{- $destinations = get "destinationType" $sync.destinationType . | append $destinations }}
-    destinationType: {{ get "destinationType" $sync.destinationType . }}
+    {{- $destinations = . | get "destinationType" $sync.destinationType | append $destinations }}
+    destinationType: {{ . | get "destinationType" $sync.destinationType }}
+    destinationPath: {{ . | get "destinationPath" "" }}
     sourceName: {{ .source }}
-    {{- $sources = get "sourceType" $main.type . | append $sources }}
-    sourceType: {{ get "sourceType" $main.type . }}
-    {{- with get "schedule" "" . }}
+    {{- $sources = . | get "sourceType" $main.type | append $sources }}
+    sourceType: {{ . | get "sourceType" $main.type }}
+    sourcePath: {{ . | get "sourcePath" "" }}
+    {{- with . | get "schedule" "" }}
     schedule: {{ . }}
     {{- end }}
+    nameSuffix: {{ . | get "nameSuffix" "custom" }}
   {{- end }}
 
 providers:

migration/v0.43/README.md

@@ -92,6 +92,30 @@ As with all scripts in this repository `CK8S_CONFIG_PATH` is expected to be set.
 
   Example of how the network policies for the pipeline can be found on the [documentation page](https://elastisys.io/welkin/operator-manual/schema/config-properties-network-policies-config-properties-network-policies-tekton-pipeline/#pipeline).
 
+1. If Harbor is using azure object storage and you are using rclone sync, then you need to disable default sync buckets and configure them manually.
+
+    You should set `objectStorage.sync.syncDefaultBuckets: false` and add the default buckets to the list of sync buckets. Except for the harbor bucket, there you instead need to add something like this:
+
+    ```yaml
+    objectStorage:
+        sync:
+            enabled: true
+            destinationType: s3
+            syncDefaultBuckets: false
+            buckets:
+            - source: <azure-env-name>-harbor
+                sourcePath: //docker
+                destinationPath: /docker # set //docker if syncing from azure to azure
+                nameSuffix: docker
+            - source: <azure-env-name>-harbor
+                sourcePath: /backups
+                destinationPath: /backups
+                nameSuffix: backups
+            # config for the other default buckets need to be added here as well
+    ```
+
+    This is because of an issue in Harbor where it is saving image data in the path `//docker/` but rclone by default skips the `//` path. So we need to add a manual config to sync that. The default sync job would also remove the docker folder from the destination since it is skipping it in the source, that is why the default job needs to be disabled.
+
 1. Apply upgrade - _disruptive_
 
     > _Done during maintenance window._

restore/harbor/README.md

@@ -45,6 +45,31 @@ envsubst > tmp-job.yaml < restore/harbor/restore-harbor-job.yaml
 
 ## Restore job Azure
 
+### Rclone move data
+
+If you are restoring Harbor in Azure from a bucket that was previously a rclone destination where the source was not Azure, then the path for the Harbor image data is likely wrong.
+Due to an [issue with Harbor](https://github.com/distribution/distribution/issues/1247) it is storing image data with an extra `/` at the start of the path for Azure.
+But it is not the same for other object storage types, so if the data was copied from e.g. S3 then it will likely be wrong now.
+The following steps will create a rclone job that will move all of the harbor image data to the correct path with an extra `/`.
+
+```bash
+export S3_BUCKET=$(yq4 '.objectStorage.buckets.harbor' "${CK8S_CONFIG_PATH}/defaults/sc-config.yaml" )
+export AZURE_ACCOUNT=$(yq4 '.objectStorage.azure.storageAccountName' "${CK8S_CONFIG_PATH}/common-config.yaml" )
+export AZURE_KEY=$(sops -d --extract '["objectStorage"]["azure"]["storageAccountKey"]' "${CK8S_CONFIG_PATH}/secrets.yaml" )
+envsubst > tmp-rclone-job.yaml < restore/harbor/harbor-rclone-azure.yaml
+./bin/ck8s ops kubectl sc apply -f tmp-rclone-job.yaml
+./bin/ck8s ops kubectl sc wait --for=condition=complete job -n rclone harbor-restore-rclone-move --timeout=-1s
+```
+
+Clean up:
+
+```bash
+./bin/ck8s ops kubectl sc delete -f tmp-rclone-job.yaml
+rm -v tmp-rclone-job.yaml
+```
+
+### Prepare for Harbor database restore
+
 Setup the necessary job.yaml and configmap:
 
 ```bash