Skip to content

Commit

Permalink
Merge branch 'master' of github.com:flyteorg/flyte into docs/migratio…
Browse files Browse the repository at this point in the history
…n-guides-ping
  • Loading branch information
pingsutw committed Mar 5, 2024
2 parents 53516b9 + 610830c commit fb34760
Show file tree
Hide file tree
Showing 51 changed files with 2,003 additions and 694 deletions.
18 changes: 17 additions & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,35 @@ jobs:
docs:
runs-on: ubuntu-latest
steps:
- name: Fetch the code
- name: Fetch flyte code
uses: actions/checkout@v4
with:
path: "${{ github.workspace }}/flyte"
- name: Fetch flytekit code
uses: actions/checkout@v4
with:
repository: flyteorg/flytekit
path: "${{ github.workspace }}/flytekit"
- uses: conda-incubator/setup-miniconda@v3
with:
auto-update-conda: true
python-version: 3.9
- shell: bash -el {0}
working-directory: ${{ github.workspace }}/flyte
run: |
conda install -c conda-forge conda-lock
conda-lock install -n monodocs-env monodocs-environment.lock.yaml
- shell: bash -el {0}
working-directory: ${{ github.workspace }}/flyte
run: |
conda activate monodocs-env
export SETUPTOOLS_SCM_PRETEND_VERSION="2.0.0"
pip install -e ./flyteidl
- shell: bash -el {0}
working-directory: ${{ github.workspace }}/flytekit
run: |
conda activate monodocs-env
pip install -e .
conda info
conda list
conda config --show-sources
Expand All @@ -49,6 +64,7 @@ jobs:
- name: Setup Graphviz
uses: ts-graphviz/setup-graphviz@v1
- name: Build the documentation
working-directory: ${{ github.workspace }}/flyte
shell: bash -el {0}
run: |
conda activate monodocs-env
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG/CHANGELOG-v1.11.0-b1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Flyte v1.11.0-b1

Second beta release for 1.11.0.
2 changes: 1 addition & 1 deletion charts/flyte-binary/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Chart for basic single Flyte executable deployment
| configuration.auth.oidc.clientId | string | `""` | |
| configuration.auth.oidc.clientSecret | string | `""` | |
| configuration.co-pilot.image.repository | string | `"cr.flyte.org/flyteorg/flytecopilot"` | |
| configuration.co-pilot.image.tag | string | `"v1.11.0-b0"` | |
| configuration.co-pilot.image.tag | string | `"v1.11.0-b1"` | |
| configuration.database.dbname | string | `"flyte"` | |
| configuration.database.host | string | `"127.0.0.1"` | |
| configuration.database.options | string | `"sslmode=disable"` | |
Expand Down
2 changes: 1 addition & 1 deletion charts/flyte-binary/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ configuration:
# repository CoPilot sidecar image repository
repository: cr.flyte.org/flyteorg/flytecopilot # FLYTECOPILOT_IMAGE
# tag CoPilot sidecar image tag
tag: v1.11.0-b0 # FLYTECOPILOT_TAG
tag: v1.11.0-b1 # FLYTECOPILOT_TAG
# agentService Flyte Agent configuration
agentService:
defaultAgent:
Expand Down
20 changes: 12 additions & 8 deletions charts/flyte-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ helm install gateway bitnami/contour -n flyte
| cloud_events.eventsPublisher.eventTypes[0] | string | `"all"` | |
| cloud_events.eventsPublisher.topicName | string | `"arn:aws:sns:us-east-2:123456:123-my-topic"` | |
| cloud_events.type | string | `"aws"` | |
| cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"nodeSelector":{},"podAnnotations":{},"podEnv":{},"podLabels":{},"service_account_name":"flyteadmin","standaloneDeployment":false,"templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain |
| cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"nodeSelector":{},"podAnnotations":{},"podEnv":{},"podLabels":{},"resources":{},"service_account_name":"flyteadmin","standaloneDeployment":false,"templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain |
| cluster_resource_manager.config | object | `{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}}` | Configmap for ClusterResource parameters |
| cluster_resource_manager.config.cluster_resources | object | `{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}}]}],"refreshInterval":"5m","standaloneDeployment":false,"templatePath":"/etc/flyte/clusterresource/templates"}` | ClusterResource parameters Refer to the [structure](https://pkg.go.dev/github.com/lyft/[email protected]/pkg/runtime/interfaces#ClusterResourceConfig) to customize. |
| cluster_resource_manager.config.cluster_resources.refreshInterval | string | `"5m"` | How frequently to run the sync process |
Expand All @@ -70,6 +70,7 @@ helm install gateway bitnami/contour -n flyte
| cluster_resource_manager.podAnnotations | object | `{}` | Annotations for ClusterResource pods |
| cluster_resource_manager.podEnv | object | `{}` | Additional ClusterResource container environment variables |
| cluster_resource_manager.podLabels | object | `{}` | Labels for ClusterResource pods |
| cluster_resource_manager.resources | object | `{}` | Resources for ClusterResource deployment |
| cluster_resource_manager.service_account_name | string | `"flyteadmin"` | Service account name to run with |
| cluster_resource_manager.templates | list | `[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]` | Resource templates that should be applied |
| cluster_resource_manager.templates[0] | object | `{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"}` | Template for namespaces resources |
Expand All @@ -94,8 +95,8 @@ helm install gateway bitnami/contour -n flyte
| configmap.clusters.clusterConfigs | list | `[]` | |
| configmap.clusters.labelClusterMap | object | `{}` | |
| configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI |
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b0","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b1","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration |
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b1","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) |
| configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration |
| configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). |
| configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). |
Expand Down Expand Up @@ -129,7 +130,7 @@ helm install gateway bitnami/contour -n flyte
| datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
| datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
| datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment |
| datacatalog.image.tag | string | `"v1.11.0-b0"` | Docker image tag |
| datacatalog.image.tag | string | `"v1.11.0-b1"` | Docker image tag |
| datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment |
| datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods |
| datacatalog.podEnv | object | `{}` | Additional Datacatalog container environment variables |
Expand Down Expand Up @@ -164,7 +165,7 @@ helm install gateway bitnami/contour -n flyte
| flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command |
| flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | |
| flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment |
| flyteadmin.image.tag | string | `"v1.11.0-b0"` | |
| flyteadmin.image.tag | string | `"v1.11.0-b1"` | |
| flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create |
| flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment |
| flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods |
Expand Down Expand Up @@ -201,7 +202,7 @@ helm install gateway bitnami/contour -n flyte
| flyteconsole.ga.tracking_id | string | `"G-0QW4DJWJ20"` | |
| flyteconsole.image.pullPolicy | string | `"IfNotPresent"` | |
| flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment |
| flyteconsole.image.tag | string | `"v1.10.3"` | |
| flyteconsole.image.tag | string | `"v1.11.0"` | |
| flyteconsole.imagePullSecrets | list | `[]` | ImagePullSecrets to assign to the Flyteconsole deployment |
| flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment |
| flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods |
Expand All @@ -225,7 +226,7 @@ helm install gateway bitnami/contour -n flyte
| flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command |
| flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | |
| flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment |
| flytepropeller.image.tag | string | `"v1.11.0-b0"` | |
| flytepropeller.image.tag | string | `"v1.11.0-b1"` | |
| flytepropeller.manager | bool | `false` | |
| flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment |
| flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods |
Expand Down Expand Up @@ -255,7 +256,7 @@ helm install gateway bitnami/contour -n flyte
| flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files |
| flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy |
| flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment |
| flytescheduler.image.tag | string | `"v1.11.0-b0"` | Docker image tag |
| flytescheduler.image.tag | string | `"v1.11.0-b1"` | Docker image tag |
| flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment |
| flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods |
| flytescheduler.podEnv | object | `{}` | Additional Flytescheduler container environment variables |
Expand Down Expand Up @@ -289,6 +290,9 @@ helm install gateway bitnami/contour -n flyte
| storage.s3.secretKey | string | `""` | AWS IAM user secret access key to use for S3 bucket auth, only used if authType is set to accesskey |
| storage.type | string | `"sandbox"` | Sets the storage type. Supported values are sandbox, s3, gcs and custom. |
| webhook.enabled | bool | `true` | enable or disable secrets webhook |
| webhook.resources.requests.cpu | string | `"200m"` | |
| webhook.resources.requests.ephemeral-storage | string | `"500Mi"` | |
| webhook.resources.requests.memory | string | `"500Mi"` | |
| webhook.securityContext | object | `{"fsGroup":65534,"fsGroupChangePolicy":"Always","runAsNonRoot":true,"runAsUser":1001,"seLinuxOptions":{"type":"spc_t"}}` | Sets securityContext for webhook pod(s). |
| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook |
| webhook.serviceAccount | object | `{"annotations":{},"create":true,"imagePullSecrets":[]}` | Configuration for service accounts for the webhook |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ spec:
image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}"
imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}"
name: sync-cluster-resources
{{- with .Values.cluster_resource_manager.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
{{- if not .Values.cluster_resource_manager.config.cluster_resources.standaloneDeployment }}
{{- include "databaseSecret.volumeMount" . | nindent 10 }}
Expand All @@ -52,9 +56,6 @@ spec:
name: cluster-secrets
{{- end }}
serviceAccountName: {{ .Values.cluster_resource_manager.service_account_name }}
{{- if .Values.cluster_resource_manager.resources }}
resources: {{- toYaml .Values.cluster_resource_manager.resources | nindent 10 }}
{{- end }}
volumes: {{- include "databaseSecret.volume" . | nindent 8 }}
- configMap:
name: clusterresource-template
Expand Down
3 changes: 3 additions & 0 deletions charts/flyte-core/templates/propeller/webhook.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,9 @@ spec:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
{{- with .Values.webhook.resources }}
resources: {{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: config-volume
mountPath: /etc/flyte/config
Expand Down
Loading

0 comments on commit fb34760

Please sign in to comment.