Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for sparkoperator webhook #284

Merged
merged 3 commits into from
Apr 21, 2020
Merged

Add support for sparkoperator webhook #284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
20a2972
Update spark-operator.yaml
akhurana001 Apr 20, 2020
85ebd58
Add RBAC for sparkoperator
Apr 20, 2020
f61a1a2
Webhook Support
Apr 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 56 additions & 3 deletions deployment/sandbox/flyte_generated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,7 @@ rules:
resources:
- services
- configmaps
- secrets
verbs:
- create
- get
Expand Down Expand Up @@ -880,6 +881,20 @@ spec:
app: contour
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
name: spark-webhook
namespace: sparkoperator
spec:
ports:
- name: webhook
port: 443
targetPort: 8080
selector:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1beta1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need any plugin side changes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, The plugin is already on v1beta1 : https://github.com/lyft/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/spark.go#L19

---
apiVersion: apps/v1
kind: Deployment
metadata:
Expand Down Expand Up @@ -1347,15 +1362,15 @@ kind: Deployment
metadata:
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1alpha1
app.kubernetes.io/version: v2.4.0-v1beta1
name: sparkoperator
namespace: sparkoperator
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1alpha1
app.kubernetes.io/version: v2.4.0-v1beta1
strategy:
type: Recreate
template:
Expand All @@ -1366,7 +1381,7 @@ spec:
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1alpha1
app.kubernetes.io/version: v2.4.0-v1beta1
spec:
containers:
- args:
Expand All @@ -1377,14 +1392,24 @@ spec:
- '-metrics-prefix=service:'
- -metrics-labels=task_name
- -metrics-labels=workflow_name
- -enable-webhook=true
- -webhook-svc-namespace=sparkoperator
command:
- /usr/bin/spark-operator
image: gcr.io/spark-operator/spark-operator:v2.4.0-v1beta1-0.9.0
imagePullPolicy: Always
name: sparkoperator-unknown
ports:
- containerPort: 10254
- containerPort: 8080
volumeMounts:
- mountPath: /etc/webhook-certs
name: webhook-certs
serviceAccountName: sparkoperator
volumes:
- name: webhook-certs
secret:
secretName: spark-webhook-certs
---
apiVersion: apps/v1
kind: StatefulSet
Expand Down Expand Up @@ -1483,6 +1508,34 @@ spec:
name: config-volume
schedule: '*/1 * * * *'
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1beta1
name: sparkoperator-init
namespace: sparkoperator
spec:
backoffLimit: 3
template:
metadata:
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1beta1
spec:
containers:
- command:
- /usr/bin/gencerts.sh
- --namespace
- sparkoperator
- -p
image: gcr.io/spark-operator/spark-operator:v2.4.0-v1beta1-0.9.0
imagePullPolicy: IfNotPresent
name: main
restartPolicy: Never
serviceAccountName: sparkoperator
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
Expand Down
16 changes: 13 additions & 3 deletions kustomize/base/operators/spark/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ metadata:
namespace: sparkoperator
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1alpha1
app.kubernetes.io/version: v2.4.0-v1beta1
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1alpha1
app.kubernetes.io/version: v2.4.0-v1beta1
strategy:
type: Recreate
template:
Expand All @@ -22,16 +22,24 @@ spec:
prometheus.io/path: "/metrics"
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1alpha1
app.kubernetes.io/version: v2.4.0-v1beta1
spec:
serviceAccountName: sparkoperator
volumes:
- name: webhook-certs
secret:
secretName: spark-webhook-certs
containers:
- name: sparkoperator-unknown
image: gcr.io/spark-operator/spark-operator:v2.4.0-v1beta1-0.9.0
volumeMounts:
- name: webhook-certs
mountPath: /etc/webhook-certs
imagePullPolicy: Always
command: ["/usr/bin/spark-operator"]
ports:
- containerPort: 10254
- containerPort: 8080
args:
- -logtostderr
- -v=2
Expand All @@ -40,3 +48,5 @@ spec:
- "-metrics-prefix=service:"
- -metrics-labels=task_name
- -metrics-labels=workflow_name
- -enable-webhook=true
- -webhook-svc-namespace=sparkoperator
1 change: 1 addition & 0 deletions kustomize/base/operators/spark/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ resources:
- spark-operator.yaml
- spark-dummy-crd.yaml
- deployment.yaml
- webhook.yaml
2 changes: 1 addition & 1 deletion kustomize/base/operators/spark/spark-operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ rules:
resources: ["pods"]
verbs: ["*"]
- apiGroups: [""]
resources: ["services", "configmaps"]
resources: ["services", "configmaps", "secrets"]
verbs: ["create", "get", "delete"]
- apiGroups: ["extensions"]
resources: ["ingresses"]
Expand Down
37 changes: 37 additions & 0 deletions kustomize/base/operators/spark/webhook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
apiVersion: batch/v1
kind: Job
metadata:
name: sparkoperator-init
namespace: sparkoperator
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1beta1
spec:
backoffLimit: 3
template:
metadata:
labels:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1beta1
spec:
serviceAccountName: sparkoperator
restartPolicy: Never
containers:
- name: main
image: gcr.io/spark-operator/spark-operator:v2.4.0-v1beta1-0.9.0
imagePullPolicy: IfNotPresent
command: ["/usr/bin/gencerts.sh","--namespace", "sparkoperator", "-p"]
---
kind: Service
apiVersion: v1
metadata:
name: spark-webhook
namespace: sparkoperator
spec:
ports:
- port: 443
targetPort: 8080
name: webhook
selector:
app.kubernetes.io/name: sparkoperator
app.kubernetes.io/version: v2.4.0-v1beta1