Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K8s: Add default values for multiple nodes platform and version #2543

Merged
merged 1 commit into from
Dec 28, 2024
Merged

K8s: Add default values for multiple nodes platform and version #2543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -948,15 +948,15 @@ chart_test_autoscaling_deployment:
./tests/charts/make/chart_test.sh DeploymentAutoscaling

chart_test_autoscaling_job_https:
PLATFORMS=$(PLATFORMS) TEST_EXISTING_KEDA=true RELEASE_NAME=selenium CHART_ENABLE_BASIC_AUTH=true SELENIUM_GRID_MONITORING=false TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) \
PLATFORMS=$(PLATFORMS) TEST_EXISTING_KEDA=true RELEASE_NAME=selenium CHART_ENABLE_BASIC_AUTH=true SELENIUM_GRID_MONITORING=false TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) TEST_MULTIPLE_PLATFORMS=true \
SECURE_CONNECTION_SERVER=true SELENIUM_GRID_PROTOCOL=https SELENIUM_GRID_PORT=443 SUB_PATH=/ \
MAX_SESSIONS_FIREFOX=1 MAX_SESSIONS_EDGE=2 MAX_SESSIONS_CHROME=3 TEST_NAME_OVERRIDE=true \
VERSION=$(TAG_VERSION) VIDEO_TAG=$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) KEDA_BASED_NAME=$(KEDA_BASED_NAME) KEDA_BASED_TAG=$(KEDA_BASED_TAG) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) BASE_VERSION=$(BASE_VERSION) EXTERNAL_UPLOADER_CONFIG=true \
TEMPLATE_OUTPUT_FILENAME="k8s_prefixSelenium_basicAuth_secureServer_autoScaling_scaledJob_existingKEDA.yaml" \
./tests/charts/make/chart_test.sh JobAutoscaling

chart_test_autoscaling_job_hostname:
PLATFORMS=$(PLATFORMS) CHART_ENABLE_TRACING=true CHART_ENABLE_BASIC_AUTH=true BASIC_AUTH_EMBEDDED_URL=true TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) \
PLATFORMS=$(PLATFORMS) CHART_ENABLE_TRACING=true CHART_ENABLE_BASIC_AUTH=true BASIC_AUTH_EMBEDDED_URL=true TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) TEST_MULTIPLE_PLATFORMS=true \
SECURE_INGRESS_ONLY_DEFAULT=true SECURE_USE_EXTERNAL_CERT=true SELENIUM_GRID_PROTOCOL=https SELENIUM_GRID_HOST=$$(hostname -I | cut -d' ' -f1) SELENIUM_GRID_PORT=443 \
VERSION=$(TAG_VERSION) VIDEO_TAG=$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) KEDA_BASED_NAME=$(KEDA_BASED_NAME) KEDA_BASED_TAG=$(KEDA_BASED_TAG) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) BASE_VERSION=$(BASE_VERSION) \
TEMPLATE_OUTPUT_FILENAME="k8s_enableTracing_basicAuth_secureIngress_externalCerts_ingressPublicIP_autoScaling_originKEDA_scaledJob_subPath.yaml" \
Expand All @@ -977,7 +977,7 @@ chart_test_autoscaling_job:
./tests/charts/make/chart_test.sh JobAutoscaling

chart_test_autoscaling_playwright_connect_grid:
PLATFORMS=$(PLATFORMS) CHART_FULL_DISTRIBUTED_MODE=true CHART_ENABLE_BASIC_AUTH=true TEST_EXTERNAL_DATASTORE=redis MATRIX_TESTS=CDPTests TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) \
PLATFORMS=$(PLATFORMS) CHART_FULL_DISTRIBUTED_MODE=true CHART_ENABLE_BASIC_AUTH=true TEST_EXTERNAL_DATASTORE=redis MATRIX_TESTS=CDPTests TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) TEST_MULTIPLE_PLATFORMS=true \
BASIC_AUTH_USERNAME=docker-selenium BASIC_AUTH_PASSWORD=2NMI4jdBi6k7bENoeUfV25295VvzwAE9chM24a+2VL95uOHozo \
SECURE_INGRESS_ONLY_DEFAULT=true SECURE_USE_EXTERNAL_CERT=true SELENIUM_GRID_PROTOCOL=https SELENIUM_GRID_HOST=$$(hostname -I | cut -d' ' -f1) SELENIUM_GRID_PORT=443 \
VERSION=$(TAG_VERSION) VIDEO_TAG=$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) KEDA_BASED_NAME=$(KEDA_BASED_NAME) KEDA_BASED_TAG=$(KEDA_BASED_TAG) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) BASE_VERSION=$(BASE_VERSION) \
Expand All @@ -993,7 +993,7 @@ test_k8s_autoscaling_job_count_strategy_default_with_node_max_sessions:
make test_k8s_autoscaling_job_count_strategy_default

test_k8s_autoscaling_job_count_strategy_default:
MATRIX_TESTS=$(or $(MATRIX_TESTS), "AutoscalingTestsScaleUp") SCALING_STRATEGY=$(or $(SCALING_STRATEGY), "default") \
MATRIX_TESTS=$(or $(MATRIX_TESTS), "AutoscalingTestsScaleUp") SCALING_STRATEGY=$(or $(SCALING_STRATEGY), "default") TEST_MULTIPLE_PLATFORMS=true \
PLATFORMS=$(PLATFORMS) RELEASE_NAME=selenium TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) SELENIUM_GRID_PROTOCOL=http SELENIUM_GRID_HOST=localhost SELENIUM_GRID_PORT=80 \
SELENIUM_GRID_MONITORING=false CLEAR_POD_HISTORY=true SET_MAX_REPLICAS=100 ENABLE_VIDEO_RECORDER=false \
VERSION=$(TAG_VERSION) VIDEO_TAG=$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) KEDA_BASED_NAME=$(KEDA_BASED_NAME) KEDA_BASED_TAG=$(KEDA_BASED_TAG) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) BASE_VERSION=$(BASE_VERSION) \
Expand All @@ -1008,7 +1008,7 @@ test_k8s_autoscaling_deployment_count_with_node_max_sessions:
make test_k8s_autoscaling_deployment_count

test_k8s_autoscaling_deployment_count:
MATRIX_TESTS=$(or $(MATRIX_TESTS), "AutoscalingTestsScaleUp") \
MATRIX_TESTS=$(or $(MATRIX_TESTS), "AutoscalingTestsScaleUp") TEST_MULTIPLE_PLATFORMS=true \
PLATFORMS=$(PLATFORMS) RELEASE_NAME=selenium TEST_PATCHED_KEDA=$(TEST_PATCHED_KEDA) SELENIUM_GRID_PROTOCOL=http SELENIUM_GRID_HOST=localhost SELENIUM_GRID_PORT=80 \
SELENIUM_GRID_MONITORING=false CLEAR_POD_HISTORY=true SET_MAX_REPLICAS=100 ENABLE_VIDEO_RECORDER=false \
VERSION=$(TAG_VERSION) VIDEO_TAG=$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) KEDA_BASED_NAME=$(KEDA_BASED_NAME) KEDA_BASED_TAG=$(KEDA_BASED_TAG) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) BASE_VERSION=$(BASE_VERSION) \
Expand Down
24 changes: 12 additions & 12 deletions charts/selenium-grid/CONFIGURATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -423,10 +423,10 @@ A Helm chart for creating a Selenium Grid Server in Kubernetes
| chromeNode.scaledOptions | string | `nil` | Override the scaled options for chrome nodes |
| chromeNode.scaledJobOptions | string | `nil` | Override the scaledJobOptions for chrome nodes |
| chromeNode.scaledObjectOptions | string | `nil` | Override the scaledObjectOptions for chrome nodes |
| chromeNode.hpa.browserName | string | `"chrome"` | browserName from the capability |
| chromeNode.hpa.browserName | string | `"chrome"` | browserName should match with Node stereotype and request capability is scaled by this scaler |
| chromeNode.hpa.sessionBrowserName | string | `"chrome"` | sessionBrowserName if the browserName is different from the sessionBrowserName |
| chromeNode.hpa.browserVersion | string | `""` | browserVersion from the capability |
| chromeNode.hpa.platformName | string | `"Linux"` | platformName from the capability |
| chromeNode.hpa.browserVersion | string | `""` | browserVersion should match with Node stereotype and request capability is scaled by this scaler |
| chromeNode.hpa.platformName | string | `""` | platformName should match with Node stereotype and request capability is scaled by this scaler |
| chromeNode.hpa.unsafeSsl | string | `"{{ template \"seleniumGrid.graphqlURL.unsafeSsl\" . }}"` | Skip check SSL when connecting to the Graphql endpoint |
| chromeNode.initContainers | list | `[]` | It is used to add initContainers in the same pod of the browser node. It should be set using the --set-json option |
| chromeNode.sidecars | list | `[]` | It is used to add sidecars proxy in the same pod of the browser node. It means it will add a new container to the deployment itself. It should be set using the --set-json option |
Expand Down Expand Up @@ -475,10 +475,10 @@ A Helm chart for creating a Selenium Grid Server in Kubernetes
| firefoxNode.scaledOptions | string | `nil` | Override the scaled options for firefox nodes |
| firefoxNode.scaledJobOptions | string | `nil` | Override the scaledJobOptions for firefox nodes |
| firefoxNode.scaledObjectOptions | string | `nil` | Override the scaledObjectOptions for firefox nodes |
| firefoxNode.hpa.browserName | string | `"firefox"` | browserName from the capability |
| firefoxNode.hpa.browserName | string | `"firefox"` | browserName should match with Node stereotype and request capability is scaled by this scaler |
| firefoxNode.hpa.sessionBrowserName | string | `"firefox"` | sessionBrowserName if the browserName is different from the sessionBrowserName |
| firefoxNode.hpa.browserVersion | string | `""` | browserVersion from the capability |
| firefoxNode.hpa.platformName | string | `"Linux"` | platformName from the capability |
| firefoxNode.hpa.browserVersion | string | `""` | browserVersion should match with Node stereotype and request capability is scaled by this scaler |
| firefoxNode.hpa.platformName | string | `""` | platformName should match with Node stereotype and request capability is scaled by this scaler |
| firefoxNode.hpa.unsafeSsl | string | `"{{ template \"seleniumGrid.graphqlURL.unsafeSsl\" . }}"` | Skip check SSL when connecting to the Graphql endpoint |
| firefoxNode.initContainers | list | `[]` | It is used to add initContainers in the same pod of the browser node. It should be set using the --set-json option |
| firefoxNode.sidecars | list | `[]` | It is used to add sidecars proxy in the same pod of the browser node. It means it will add a new container to the deployment itself. It should be set using the --set-json option |
Expand Down Expand Up @@ -527,10 +527,10 @@ A Helm chart for creating a Selenium Grid Server in Kubernetes
| edgeNode.scaledOptions | string | `nil` | Override the scaled options for edge nodes |
| edgeNode.scaledJobOptions | string | `nil` | Override the scaledJobOptions for edge nodes |
| edgeNode.scaledObjectOptions | string | `nil` | Override the scaledObjectOptions for edge nodes |
| edgeNode.hpa.browserName | string | `"MicrosoftEdge"` | browserName from the capability |
| edgeNode.hpa.browserName | string | `"MicrosoftEdge"` | browserName should match with Node stereotype and request capability is scaled by this scaler |
| edgeNode.hpa.sessionBrowserName | string | `"msedge"` | sessionBrowserName if the browserName is different from the sessionBrowserName |
| edgeNode.hpa.browserVersion | string | `""` | browserVersion from the capability |
| edgeNode.hpa.platformName | string | `"Linux"` | platformName from the capability |
| edgeNode.hpa.browserVersion | string | `""` | browserVersion should match with Node stereotype and request capability is scaled by this scaler |
| edgeNode.hpa.platformName | string | `""` | platformName should match with Node stereotype and request capability is scaled by this scaler |
| edgeNode.hpa.unsafeSsl | string | `"{{ template \"seleniumGrid.graphqlURL.unsafeSsl\" . }}"` | Skip check SSL when connecting to the Graphql endpoint |
| edgeNode.initContainers | list | `[]` | It is used to add initContainers in the same pod of the browser node. It should be set using the --set-json option |
| edgeNode.sidecars | list | `[]` | It is used to add sidecars proxy in the same pod of the browser node. It means it will add a new container to the deployment itself. It should be set using the --set-json option |
Expand Down Expand Up @@ -579,10 +579,10 @@ A Helm chart for creating a Selenium Grid Server in Kubernetes
| relayNode.scaledOptions | string | `nil` | Override the scaled options for relay nodes |
| relayNode.scaledJobOptions | string | `nil` | Override the scaledJobOptions for relay nodes |
| relayNode.scaledObjectOptions | string | `nil` | Override the scaledObjectOptions for relay nodes |
| relayNode.hpa.browserName | string | `"chrome"` | browserName from the capability |
| relayNode.hpa.browserName | string | `"chrome"` | browserName should match with Node stereotype and request capability is scaled by this scaler |
| relayNode.hpa.sessionBrowserName | string | `""` | sessionBrowserName if the browserName is different from the sessionBrowserName |
| relayNode.hpa.platformName | string | `"Android"` | platformName from the capability |
| relayNode.hpa.browserVersion | string | `""` | browserVersion from the capability |
| relayNode.hpa.browserVersion | string | `""` | browserVersion should match with Node stereotype and request capability is scaled by this scaler |
| relayNode.hpa.platformName | string | `"Android"` | platformName should match with Node stereotype and request capability is scaled by this scaler |
| relayNode.hpa.unsafeSsl | string | `"{{ template \"seleniumGrid.graphqlURL.unsafeSsl\" . }}"` | Skip check SSL when connecting to the Graphql endpoint |
| relayNode.initContainers | list | `[]` | It is used to add initContainers in the same pod of the browser node. It should be set using the --set-json option |
| relayNode.sidecars | list | `[]` | It is used to add sidecars proxy in the same pod of the browser node. It means it will add a new container to the deployment itself. It should be set using the --set-json option |
Expand Down
136 changes: 136 additions & 0 deletions charts/selenium-grid/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This chart enables the creation of a Selenium Grid Server in Kubernetes.
* [Settings common for both `job` and `deployment` scalingType](#settings-common-for-both-job-and-deployment-scalingtype)
* [Settings when scalingType with `deployment`](#settings-when-scalingtype-with-deployment-)
* [Settings when scalingType with `job`](#settings-when-scalingtype-with-job)
* [Scaler trigger configuration](#scaler-trigger-configuration)
* [Settings fixed-sized thread pool for the Distributor to create new sessions](#settings-fixed-sized-thread-pool-for-the-distributor-to-create-new-sessions)
* [Updating Selenium-Grid release](#updating-selenium-grid-release)
* [Uninstalling Selenium Grid release](#uninstalling-selenium-grid-release)
Expand Down Expand Up @@ -210,6 +211,141 @@ autoscaling:

Settings that KEDA [ScaledJob spec](https://keda.sh/docs/latest/concepts/scaling-jobs/#scaledjob-spec) supports can be set via `autoscaling.scaledJobOptions`.

Expected that with default configuration in KEDA resource, autoscaling behavior should be correct. Hence, in chart values, we keep the config key `autoscaling.scaledJobOptions.scalingStrategy.strategy` is `default`.

### Scaler trigger configuration

From KEDA core `v2.16.1+`, the trigger metadata `browserVersion`, `platformName` is recommended to be set explicitly to have the correct scaling behavior (especially when your Grid includes autoscaling Nodes, non-autoscaling Nodes, relay Nodes, etc.). Besides that, in client binding, it is also recommended to set the `browserVersion`, `platformName` to align with the trigger metadata. Please see below examples for more details.

Understand list trigger parameters

- `url` - Graphql url of your Selenium Grid. If endpoint requires authentication, you can use `TriggerAuthentication` to provide the credentials instead of embedding in the URL.
- `browserName` - browserName should match with Node stereotype and request capability is scaled by this scaler. (Default: ``, Optional)
- `sessionBrowserName` - sessionBrowserName if the browserName is different from the sessionBrowserName. (Default: ``, Optional)
- `browserVersion` - browserVersion should match with Node stereotype and request capability is scaled by this scaler. (Default: ``, Optional)
- `platformName` - platformName should match with Node stereotype and request capability is scaled by this scaler. (Default: ``, Optional)
- `unsafeSsl` - Skip certificate validation when connecting over HTTPS. (Default: `false`, Optional)
- `activationThreshold` - Target value for activating the scaler. Learn more about activation [here](./../concepts/scaling-deployments.md#activating-and-scaling-thresholds). (Default: `0`, Optional)
- `nodeMaxSessions` - Number of maximum sessions that can run in parallel on a Node. Update this parameter align with node config `--max-sessions` (`SE_NODE_MAX_SESSIONS`) to have the correct scaling behavior. (Default: `1`, Optional).

Understand list trigger authentication

- `username` - Username for basic authentication in GraphQL endpoint instead of embedding in the URL. (Optional)
- `password` - Password for basic authentication in GraphQL endpoint instead of embedding in the URL. (Optional)
- `authType` - Type of authentication to be used. This can be set to `Bearer` or `OAuth2` in case Selenium Grid behind an Ingress proxy with other authentication types. (Optional)
- `accessToken` - Access token. This is required when `authType` is set a value. (Optional)

In each Node, trigger parameters value will be set under config key `hpa`. In template, those will be added spec of ScaledObject/ScaledJob.

In chart values, by default, `browserName`, `sessionBrowserName` are set for corresponding node browser. Parameters `browserVersion`, `platformName` are not set, leave them as empty by default. The default scaler metadata looks like

```yaml
triggers:
- type: selenium-grid
metadata:
url: 'http://selenium-hub:4444/graphql'
browserName: 'chrome'
browserVersion: ''
platformName: ''
```

In this case, the scaler will be triggered by below request (example in Python client, common use case that most users get started)

```python
options = ChromeOptions()
driver = webdriver.Remote(options=options, command_executor=SELENIUM_GRID_URL)
```

With above script, the request is sent to Grid. Via GraphQL response, it looks like

```json
{
"data": {
"grid": {
"sessionCount": 0,
"maxSession": 0,
"totalSlots": 0
},
"nodesInfo": {
"nodes": []
},
"sessionsInfo": {
"sessionQueueRequests": [
"{\"browserName\": \"chrome\"}"
]
}
}
}
```

Scaler will trigger to scale up the Node with stereotypes matched to pick up the request in the queue. Via GraphQL response, it looks like

```json
{
"data": {
"grid": {
"sessionCount": 0,
"maxSession": 1,
"totalSlots": 1
},
"nodesInfo": {
"nodes": [
{
"id": "UUID",
"status": "UP",
"sessionCount": 0,
"maxSession": 1,
"slotCount": 1,
"stereotypes": "[{\"slots\": 1, \"stereotype\": {\"browserName\": \"chrome\", \"browserVersion\": \"\", \"platformName\": \"\"}}]",
"sessions": []
}
]
},
"sessionsInfo": {
"sessionQueueRequests": [
"{\"browserName\": \"chrome\"}"
]
}
}
}
```

In Node deployment spec, there is environment variable `SE_NODE_BROWSER_VERSION` which is able to unset `browserVersion` in Node stereotypes (it is setting short browser build number by default e.g `131.0`) or any custom value is up to you, which is expected to match with the request capabilities in queue and scaler trigger metadata.
Similarly, `SE_NODE_PLATFORM_NAME` is used to unset the `platformName` in Node stereotypes if needed. Noted, update to newer image tag if these 2 env variables doesn't take effect for you.

For another example, where your Grid with multiple scalers have different metadata, one of them looks like

```yaml
triggers:
- type: selenium-grid
metadata:
url: 'http://selenium-hub:4444/graphql'
browserName: 'chrome'
browserVersion: '131.0'
platformName: 'Linux'
```


The request to trigger this corresponds to the following Python script

```python
options = ChromeOptions()
options.set_capability('platformName', 'Linux')
options.set_capability('browserVersion', '131.0')
driver = webdriver.Remote(options=options, command_executor=SELENIUM_GRID_URL)
```

### Define multiple scalers with different trigger parameters.
When deploying the chart, you can define multiple scalers with different trigger parameters to scale up different Node stereotypes against different request capabilities.

Under config key `crossBrowsers`, in corresponding browser node, you can define array of item with structure same as that node, via `nameOverride` to set unique name for each scaler to avoid resources collision.

For example [multiple-nodes-platform.yaml](./multiple-nodes-platform.yaml) file, it defines 2 scalers per browser node to scale against requests with and without `platformName` capability.

For example [multiple-nodes-platform-version.yaml](./multiple-nodes-platform-version.yaml) file, it defines multiple scalers with `platformName: 'Linux'` and last few previous stable versions per browser node to scale against requests with `browserVersion` and `platformName` capabilities.

While deploying the chart, you can quickly use these extra values files by passing the file via `--values` flag to apply.

### Settings fixed-sized thread pool for the Distributor to create new sessions

When enabling autoscaling, the Distributor might be under a high workload with parallelism tests, which are many requests incoming and nodes scaling up simultaneously. (Refer to: [SeleniumHQ/selenium#13723](https://github.com/SeleniumHQ/selenium/issues/13723)).
Expand Down
Loading
Loading