Skip to content
This repository has been archived by the owner on Nov 14, 2023. It is now read-only.

Commit

Permalink
helm-chart: make sure database migrations are only run by one process (
Browse files Browse the repository at this point in the history
…cvat-ai#6780)

To do that, remove the running of migrations from the server pod(s), and
add a Kubernetes job to do it instead.

This will prevent Django from corrupting the database when two or more
instances are started up at the same time.
  • Loading branch information
SpecLad authored and mikhail-treskin committed Oct 25, 2023
1 parent b5665cf commit 6c9ee2f
Show file tree
Hide file tree
Showing 5 changed files with 86 additions and 3 deletions.
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- TDB

### Changed
- TDB
- \[Helm\] Database migrations now run in a separate job instead of the server pod,
in order to avoid data corruption when multiple replicas of the server are used
(<https://github.com/opencv/cvat/pull/6780>)

### Deprecated
- TDB
Expand Down
2 changes: 1 addition & 1 deletion helm-chart/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.9.1
version: 0.10.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
Expand Down
72 changes: 72 additions & 0 deletions helm-chart/templates/cvat_backend/initializer/job.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
apiVersion: batch/v1
kind: Job
metadata:
# Jobs are mostly immutable, so Helm can't update them when the user upgrades
# a release. Work around this by including the revision in the job name, so that
# every upgrade creates a new job.
# See also <https://github.com/helm/helm/issues/7082>.
name: {{ .Release.Name }}-backend-initializer-r{{ .Release.Revision }}
namespace: {{ .Release.Namespace }}
labels:
app: cvat-app
tier: backend
component: initializer
{{- include "cvat.labels" . | nindent 4 }}
{{- with .Values.cvat.backend.initializer.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.cvat.backend.initializer.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
template:
metadata:
labels:
app: cvat-app
tier: backend
component: initializer
{{- include "cvat.labels" . | nindent 8 }}
{{- with .Values.cvat.backend.initializer.labels }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.cvat.backend.initializer.annotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
containers:
- name: cvat-app-backend-initializer-container
image: {{ .Values.cvat.backend.image }}:{{ .Values.cvat.backend.tag }}
imagePullPolicy: {{ .Values.cvat.backend.imagePullPolicy }}
{{- with .Values.cvat.backend.initializer.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
args: ["init"]
env:
{{ include "cvat.sharedBackendEnv" . | indent 10 }}
{{- with .Values.cvat.backend.initializer.additionalEnv }}
{{- toYaml . | nindent 10 }}
{{- end }}
{{- with .Values.cvat.backend.initializer.additionalVolumeMounts }}
volumeMounts:
{{- toYaml . | nindent 10 }}
{{- end }}
restartPolicy: OnFailure
{{- with .Values.cvat.backend.initializer.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.cvat.backend.initializer.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.cvat.backend.initializer.additionalVolumes }}
volumes:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
2 changes: 1 addition & 1 deletion helm-chart/templates/cvat_backend/server/deployment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ spec:
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
args: ["init", "run", "server"]
args: ["run", "server"]
env:
- name: ALLOWED_HOSTS
value: {{ .Values.cvat.backend.server.envs.ALLOWED_HOSTS | squote}}
Expand Down
9 changes: 9 additions & 0 deletions helm-chart/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,15 @@ fullnameOverride: ""

cvat:
backend:
initializer:
labels: {}
annotations: {}
resources: {}
affinity: {}
tolerations: []
additionalEnv: []
additionalVolumes: []
additionalVolumeMounts: []
server:
replicas: 1
labels: {}
Expand Down

0 comments on commit 6c9ee2f

Please sign in to comment.