Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Flyte Helm Chart #916

Merged
merged 93 commits into from
May 29, 2021
Merged
Changes from 1 commit
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
8f1ea41
update docs theme (#876)
cosmicBboy Mar 30, 2021
be8db59
Helm chart for Flyte - initial commit
rstanevich Oct 13, 2020
4908da1
remove new lines
rstanevich Oct 13, 2020
90a7422
Added autogenerated Helm docs
rstanevich Oct 14, 2020
565f0b4
improve templates
rstanevich Nov 9, 2020
8ec2cab
Support gRPC loadbalncing for EKS with AWS ALB
rstanevich Feb 2, 2021
249711f
Use external chart dependencies for contour and spark-operator
sbrunk Feb 23, 2021
6ca7ef6
Helm - Update task plugin config
sbrunk Mar 19, 2021
11402cd
Helm - Updating Spark config to force default credentials chain for AWS
sbrunk Mar 19, 2021
635fe7f
Helm - Update contour GRPC annotation
sbrunk Mar 19, 2021
fdd49e8
Helm - Enabling Event version 1 by default
sbrunk Mar 19, 2021
ebb5e40
Helm - Improve getting started experience
sbrunk Mar 19, 2021
f7bcff2
Helm - Exponential error back-off in case of system errors
sbrunk Mar 19, 2021
ddab0df
Helm - Trim sandbox
sbrunk Mar 20, 2021
bacb132
Helm - Update default RAM Request for tasks in sandbox
sbrunk Mar 20, 2021
08a4581
Helm - Update Admin task resources
sbrunk Mar 20, 2021
ddbdaeb
Helm - Add kubernetes dashboard and update deployments
sbrunk Mar 20, 2021
cfcd263
Helm - Update EKS Ingress
sbrunk Mar 31, 2021
31a657f
Move GRPC annotations from ingress to services where possible
sbrunk Apr 1, 2021
c30aad9
Improve installation docs
sbrunk Apr 1, 2021
a143dcb
Downgrade Ingress API version for K8S 1.18 compatibility
sbrunk Apr 12, 2021
682aa27
Add tls option for ingress
sbrunk Apr 13, 2021
b4158be
Remove redundant default values from EKS config
sbrunk Apr 14, 2021
4c8c7a5
Simplify Helm storage configuration
sbrunk Apr 14, 2021
d2e9321
[wip] Helm components (#921)
kumare3 Apr 20, 2021
643a8b2
Working helm deployment
kumare3 Apr 20, 2021
c7d0470
more docs
kumare3 Apr 20, 2021
040a7f6
Add references to flyteadmin config
katrogan Apr 22, 2021
3a14808
Improved value.yaml docs
kumare3 Apr 21, 2021
6235ead
updated yaml
kumare3 Apr 21, 2021
1cd0ce8
Updated working basic eks config
kumare3 Apr 23, 2021
35d1790
Improve installation docs and bump versions
sbrunk Apr 27, 2021
919a2a7
Fix service account annotation structure in templates
sbrunk Apr 27, 2021
e2bda95
Fully working
kumare3 Apr 26, 2021
7dd63fa
Update image tags, fix customDataConfig and expose defaultIamRole as …
EngHabu May 6, 2021
4175f26
Allow remote data config for admin
EngHabu May 6, 2021
3346a5a
Use bet version of ingress (#988)
juandiegopalomino May 6, 2021
54c145c
Add auth services to ingress
EngHabu May 6, 2021
9c17ceb
Support imagePullSecrets for service accounts
EngHabu May 7, 2021
e89b23b
Update Console version
EngHabu May 10, 2021
889cd66
Add Auth support to Helm chart
sbrunk May 7, 2021
f74ee3f
update flytepropeller
EngHabu May 11, 2021
b6247b1
Add Pod Webhook deployment to Helm chart (#972)
sbrunk May 11, 2021
4bd12c3
dark theme updates (#883)
cosmicBboy Mar 31, 2021
6b8ed62
Release automation addedd for flyte definition (#835)
yindia Apr 1, 2021
0781077
Milestone 0.12.0 (#888)
flyte-bot Apr 1, 2021
d84ade5
Update index.rst (#884)
SandraGH5 Apr 2, 2021
aca3089
Dashboard updates (#886)
kumare3 Apr 3, 2021
bea33d0
Add init container that will pre-create buckets in minio (#905)
jeevb Apr 9, 2021
7604f90
Add LF AI & Data Logo to README (#902)
samhita-alla Apr 11, 2021
3a7c646
Explaining the state machine used by FlytePropeller and Flyte (#903)
kumare3 Apr 12, 2021
6db935a
Fix default eks storage config (#907)
EngHabu Apr 13, 2021
76e94e5
Remove flyteadmin config options which now have defaults (#917)
Apr 15, 2021
64e6e0e
Contribution Guide (#922)
samhita-alla Apr 20, 2021
a30e703
explicitly grant flyteworkflow finalizer permissions to flytepropelle…
akumor Apr 22, 2021
8420bc4
Add dynamic running state to node state machine (#929)
Apr 22, 2021
e7c2893
Updated Boilerplate Code (#932)
samhita-alla Apr 27, 2021
89ca2e7
Added goreleaser for publishing menifest (#927)
yindia Apr 28, 2021
0248dc9
Added flytectl in sandbox (#952)
yindia Apr 28, 2021
ca95973
Modify repo name in Makefile (#953)
samhita-alla Apr 29, 2021
c168ec2
Signed-off-by: Samhita Alla aallasamhita@gmail.com (#955)
cosmicBboy Apr 30, 2021
13190b0
added tagline (#956)
samhita-alla Apr 30, 2021
bbbff78
Fix broken Kustomize link in docs (#944)
orf Apr 30, 2021
88a3174
update getting started guide (#958)
cosmicBboy Apr 30, 2021
f8a2a31
Authentication Config Examples (#970)
EngHabu May 5, 2021
719ead2
Flyte milestone v0.13 release (#979)
flyte-bot May 5, 2021
62f7ec4
Add Auth Ingress endpoints (#982)
EngHabu May 6, 2021
14cbc43
Updated Readme.md with CII Badge (#990)
kumare3 May 7, 2021
4632300
Wait for deployments to be created using rollout before checking for …
pmahindrakar-oss May 7, 2021
cad2857
update links to autogenerated flytesnacks pages (#999)
cosmicBboy May 7, 2021
1a53db9
Include troubleshooting guide in the docs (#987)
samhita-alla May 11, 2021
dc44f8d
Add data council video
EngHabu May 11, 2021
e846c27
Added tip for make start to check troubleshooting page (#1023)
pmahindrakar-oss May 12, 2021
7d3e976
Update README.md (#1022)
kumare3 May 12, 2021
3abc69a
Rename admin server config section
EngHabu May 12, 2021
1332b9e
Fix deployment indentation
EngHabu May 12, 2021
3362681
Update image versions
EngHabu May 12, 2021
2620dba
disable tls by default because of: https://github.com/helm/helm/issue…
EngHabu May 19, 2021
de6076b
default imagepullsecrets
EngHabu May 21, 2021
afcfa04
Disable SSL Redirect for GRPC
EngHabu May 21, 2021
b41f4b6
Fix spark config map
EngHabu May 24, 2021
51dd9f4
Update imagePullSecrets
EngHabu May 25, 2021
13173e0
Add make helm to generate static helm charts
EngHabu May 25, 2021
848845a
add --dependency-update to helm command
EngHabu May 25, 2021
b060572
Don't hardcode namespace in flyte-admin-auth secret
sbrunk May 25, 2021
c705978
generate helm
EngHabu May 25, 2021
32f2786
Helm template upgrade to auto-generate co-pilot configuration (#1056)
kumare3 May 27, 2021
b637c21
Install deps once
EngHabu May 28, 2021
fba8387
Install helm-docs if not found
EngHabu May 28, 2021
833f1e1
bump
EngHabu May 28, 2021
d9ea2b5
Merge branch 'master' into helm-working-branch
EngHabu May 28, 2021
6399932
Fix merge
EngHabu May 28, 2021
ff0f087
Update helm docs
EngHabu May 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update image tags, fix customDataConfig and expose defaultIamRole as …
…a template field

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
EngHabu committed May 28, 2021

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
commit 7dd63fa00e6fd135e8f8c9063b020d7607f1fa50
72 changes: 35 additions & 37 deletions helm/values-eks.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

#
# FLYTEADMIN
#
@@ -159,8 +158,6 @@ minio:
contour:
enabled: false



#
# COMMON
#
@@ -198,7 +195,6 @@ common:
# -- If using plain text you can provide the password here
pass.txt: awesomesauce


# -----------------------------------------------------
# Core dependencies that should be configured for Flyte to work on any platform
# Specifically 2 - Storage (s3, gcs etc), Production RDBMS - Aurora, CloudSQL etc
@@ -244,11 +240,6 @@ configmap:
allowedHeaders:
- "Content-Type"

task_type_whitelist:
spark:
- project: flytetester
- project: spark-workflows

task_resource_defaults:
task_resources:
defaults:
@@ -288,18 +279,18 @@ configmap:
- k8s-array
- pytorch
- athena
# - sagemaker_hyperparameter_tuning
# - sagemaker_custom_training
# - sagemaker_training
# - sagemaker_hyperparameter_tuning
# - sagemaker_custom_training
# - sagemaker_training
default-for-task-types:
container: container
sidecar: sidecar
spark: spark
container_array: k8s-array
pytorch: pytorch
hive: athena
# sagemaker_custom_training_task: sagemaker_custom_training
# sagemaker_custom_training_job_task: sagemaker_custom_training
# sagemaker_custom_training_task: sagemaker_custom_training
# sagemaker_custom_training_job_task: sagemaker_custom_training

logger:
logger:
@@ -320,7 +311,6 @@ configmap:
# -- cloudwatch log-group
cloudwatch-log-group: <LOG_GROUP_NAME>


# ----------------------------------------------------------------
# Optional Modules
# Flyte built extensions that enable various additional features in Flyte.
@@ -348,29 +338,27 @@ workflow_scheduler:
reconnectAttempts: 10
reconnectDelaySeconds: 30


# --
# Workflow notifications module is an optional dependency. Flyte uses cloud native pub-sub systems to notify users of
# various events in their workflows
workflow_notifications:
enabled: false
config:
notifications:
type: aws
region: us-east-2
publisher:
topicName: arn:aws:sns:us-east-1:<ACCOUNT-NUMBER>:flyte-notifications-topic
processor:
queueName: flyte-notifications-queue
accountId: <ACCOUNT-NUMBER>
emailer:
subject: "Flyte: {{ project }}/{{ domain }}/{{ launch_plan.name }} has '{{ phase }}'"
sender: "flyte@example.com"
body: |
Execution {{ workflow.project }}/{{ workflow.domain }}/{{ workflow.name }}/{{ name }} has {{ phase }}.
Details: https://flyte.example.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}.
{{ error }}

enabled: false
config:
notifications:
type: aws
region: us-east-2
publisher:
topicName: arn:aws:sns:us-east-1:<ACCOUNT-NUMBER>:flyte-notifications-topic
processor:
queueName: flyte-notifications-queue
accountId: <ACCOUNT-NUMBER>
emailer:
subject: "Flyte: {{ project }}/{{ domain }}/{{ launch_plan.name }} has '{{ phase }}'"
sender: "flyte@example.com"
body: |
Execution {{ workflow.project }}/{{ workflow.domain }}/{{ workflow.name }}/{{ name }} has {{ phase }}.
Details: https://flyte.example.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}.
{{ error }}

# -- Configuration for the Cluster resource manager component. This is an optional component, that enables automatic
# cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain
@@ -389,6 +377,17 @@ cluster_resource_manager:
finalizers:
- kubernetes

# -- Patch default service account
- key: aab_default_service_account
value: |
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: {{ namespace }}
annotations:
eks.amazonaws.com/role-arn: {{ defaultIamRole }}

- key: ab_project_resource_quota
value: |
apiVersion: v1
@@ -455,6 +454,8 @@ cluster_resource_manager:
metadata:
name: spark
namespace: {{ namespace }}
annotations:
eks.amazonaws.com/role-arn: {{ defaultIamRole }}

# -- Template for spark role binding
- key: af_spark_role_binding
@@ -507,7 +508,6 @@ pytorchoperator:
tf_operator:
enabled: false


# -----------------
# -- Training on AWS Sagemaker using AWS Sagemaker operator. To actually install the operator, please follow instructions [here](https://github.com/aws/amazon-sagemaker-operator-for-k8s/tree/master/hack/charts/installer/rolebased)
# Use the config section here to just enable sagemaker plugin in Flyte, after you have installed the operator using the information
@@ -518,8 +518,6 @@ sagemaker:
sagemaker:
roleArn: arn:aws:iam::<ACCOUNT_NUMBER>:role/<ROLE_FOR_SAGEMAKER>
region: us-east-2


# ---------------------------
# Flink jobs using the Flink Operator
#
8 changes: 4 additions & 4 deletions helm/values.yaml
Original file line number Diff line number Diff line change
@@ -105,7 +105,7 @@ flytepropeller:
image:
# -- Docker image for Flytepropeller deployment
repository: ghcr.io/flyteorg/flytepropeller
tag: v0.10.4
tag: v0.10.7
pullPolicy: IfNotPresent
# -- Default resources requests and limits for Flytepropeller deployment
resources:
@@ -625,17 +625,17 @@ cluster_resource_manager:
cluster_resources:
templatePath: "/etc/flyte/clusterresource/templates"
customData:
production:
- production:
- projectQuotaCpu:
value: "5"
- projectQuotaMemory:
value: "4000Mi"
staging:
- staging:
- projectQuotaCpu:
value: "2"
- projectQuotaMemory:
value: "3000Mi"
development:
- development:
- projectQuotaCpu:
value: "4"
- projectQuotaMemory: