Update ELK stack helmfiles #380
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
willgraf
commented
Oct 5, 2020
willgraf
commented
- Remove unused defaults in the helmfiles to prevent unexpected issues when upgrading versions.
- Update default arguments for consistency with other helmfiles
- Add documentation references to each helmfile.
Lots of stuff in this squashed commit: * Integration tests for cluster deployment on GKE were added. Part of this process was paring down a minimal build environment with 1) a minimal set of environmental variables, 2) a permanent service account with a minimal set of permissions, and 3) a VM environment with a minimal set of dependencies (see `.travis/install_script.sh`). * Integration testing utilizes as much production code as possible (almost total overlap). * All testing is currently done in TravisCI and has been optimized to minimize build time. * Integration tests only run on merges into master, but can be triggered at any time by placing [build-integration-tests] in a commit message. * Testing of ELK-enabled clusters is currently turned off, due to the potential for dangling resources upon cluster failure. ELK-enabled integration tests can be triggered by including both [build-integration-tests] and [test-elk] in the same commit message. * The project's pull request template now "requires" integration tests to be run, although it's up to the reviewer to enforce this requirement. Minor: * Added CONF_PATH_PREFIX variable for compatibility between production environment and testing VMs. * Replaced shopt calls with a new system for toggling ELK deployment. (ELK_DEPLOYMENT_TOGGLE = "" by default and any other value turns ELK deployment on.)
* remove chart quotes and set all image.pullPolicy to IfNotPresent * remove pullPolicy from helmfile, no need to override by default
* remove numbers and update version * bold headline in rst * copyedited for clarity and correctness * fix formatting * change dev content to note * change formatting in Usage * update license badge with correct target * update README cmd version to 1.1.0 * crop images to be the same size
* use `gomplate` for templating `hpa.yaml`. * change reference in the docs as well. * remove unused `GPU_MAX_TIMES_X` and `GPU_MAX_DIVIDED_BY_X` env vars.
… of kiosk (#295) * Change all references to vanvalenlab/kiosk to vanvalenlab/kiosk-console. * Change run command from `kiosk` to `kiosk-console`. * Update architecture diagram to include kiosk-console. * Clarify differences between DeepCell Kiosk and kiosk-console.
The previous chart version is 1.1.0, which I think was intended to be the app version instead.
* change GPU_MIN_NODES to GPU_NODE_MIN_SIZE and GPU_MAX_NODES to GPU_NODE_MAX_SIZE
* Change default value for GCP_PREDICTION_GPU_TYPE to nvidia-tesla-t4 * GPU only needs to be in limits, not requests
* remove serviceAccount default value from helmfile * update image to 0.4.1 * reduce CPU request to 50m * unstringify helmfile
* Upgrade redis-janitor image to 0.3.1 * Unstringify helmfile * Remove unused serviceAccount in helmfile
Includes dataLabel field in the backend API predict spec
* Update prometheus-operator, prometheus-adapter, and promethes-redis-exporter helm charts and remove stale default values * Relabel redis-exporter with `deployment=$QUEUE-consumer` and change key to be `$QUEUE` * Rename zip-consumer to segmentation-zip-consumer to match labels. * Using .75 instead of .9 for backoff coefficient.
* Updating openvpn means updating its documentation as well!
Catch stable branch up to master
* Update all consumers to 0.5.1. * Update model environment variables to DeepWatershed versions. * Deprecate `SAMPLE`, `RESHAPE_SIZE`, and `TF_TENSOR_DTYPE` env vars by using the `GetModelMetadata` API. * Support `TF_MAX_BATCH_SIZE` and `TF_MIN_MODEL_SIZE` for batching consumer requests to tf-serving. * Set `MAX_RETRY` to 5 by default for consumer jobs. * Set tf-serving `MAX_BATCH_SIZE` to 128. * Set consumers `TF_MAX_BATCH_SIZE` to 64 (less than actual max batch size). * Unstringify helmfiles for aesthetics. Co-Authored-By: willgraf <[email protected]> Co-Authored-By: MekWarrior <[email protected]>
…ing AOF fsync (#316) * use "no-appendfsync-on-rewrite yes" configuration setting for reduced redis downtime. * bump slave count from 2 to 3
…nt. (#314) * ignore some kubeApiServer and cAdvisor metrics using metricRelabelings * ignore large kubelet metrics Co-Authored-By: willgraf <[email protected]> Co-Authored-By: MekWarrior <[email protected]>
* Install yq in install_script.sh * awk print the last column to fix yq v2 vs v3. [build-integration-tests] Co-Authored-By: willgraf <[email protected]> Co-Authored-By: MekWarrior <[email protected]>
* create a 4 digit random number instead of a 2 digit for more entropy * set test cluster name to be 4 digits max to stay under the 18 limit. Co-Authored-By: willgraf <[email protected]> Co-Authored-By: MekWarrior <[email protected]>
* password warning about sudo. * Simplify quota instruction. * include standard storage with fine-grain access controls in bucket creation step. * more descriptive cluster name box header. * clarify Configuration option in instructions. * direct links to the google docs for quota resources. * proper formatting for menu options. * Compute Engine > VM instances menu is not necessary as a link is provided. * make GPU reference searchable in the console.
* lint helm charts and helmfiles. * move all shell scripts from patches into scripts. * gomplate all addon yaml files. * move full yaml configurations into addons. patches is for updating specs of existing resources. Co-authored-by: MekWarrior <[email protected]>
* build and run the docker container to run integration tests. * Move tests into conf/makefile * change keyfile to `GOOGLE_APPLICATION_CREDENTIALS`. * remove deprecated install_script.sh. * remove deprecated `CONF_PATH_PREFIX`.
* bucket region warning in create confirmation. * adjust the height for new warning.
[build-integration-tests] [test-elk]
[build-integration-tests] [test-elk]
[build-integration-tests] [test-elk]
|
Integration tests have successfully passed in |
willgraf
added a commit
that referenced
this pull request
Dec 16, 2020
* Support TLS traffic with cert-manager. (#357) * Fix frontend ingress issue when no hosts are provided. (#381) * Template frontend ingress annotations using `CERTIFICATE_MANAGER_ENABLED` (#383) * Create tf-serving configuration files using an initContainer. (#382) * Fix whitespace issue in tasks/Makefile.kubectl (#386) * Bump openvpn to 4.2.3 (#385) * Upgrade certificate manager to version 1.0.3 (#384) * Add screenshot of successfully created cluster to docs. (#388) * Set up an AlertManager with slack receiver support (#317) * Install procps to give access to sysctl. (#390) * Migrate CI/CD from TravisCI to GitHub Actions (#394) * Change the redis helm chart repo to bitnami (#393) * Upgrade tf-serving chart to 0.3.0 for application version 0.4.0 (#392) * Move the frontend HPA definition into the helm chart. (#395) * Move the tf-serving HPA into the helm chart. (#396) * Move redis-consumer HPA into the helm chart. (#397) * Remove deprecated and unused charts (#398) * Migrate stable helm chart repo to archived URL. (#399) * Destroy the secret and remove the key from the DNS solver SA in a new task: `gke/destroy/certificate-manager-secret` (fixes #391). * Use GCP_SERVICE_ACCOUNT for DNS resolution (#401) * Clean up docs and test them with new GitHub Action workflow (#402) * Add code-formatted filename to list of files to change (#403) * Update ELK stack helmfiles (#380) * Move the prometheus-redis-exporter script to a chart using incubator/raw. (#405) * Use `kubectl del pvc` instead of deleting all pds with the cluster name. (#406) * Update helmfile defaults for faster helm deployments. (#404) * Skip gke/destroy/node-pools during cluster teardown. (#407) * Update docs to reflect the pending 1.4.0 release. (#408) * Bump redis-consumer version to 0.8.3 (#409) * Run integration tests on all PRs to master OR if they have the commit message. (#411) * Remove helm defaults for ELK helmfiles (#413) Co-authored-by: Morgan Schwartz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.