-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[receiver/apachespark] Outline apache spark receiver component (#21971)
* [receiver/apachespark] Outline apache spark receiver component
- Loading branch information
1 parent
6f1b6b9
commit 51d1056
Showing
26 changed files
with
929 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' | ||
| change_type: new_component | ||
|
|
||
| # The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver) | ||
| component: apachesparkreceiver | ||
|
|
||
| # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). | ||
| note: adds the apachesparkreceiver metric receiver | ||
|
|
||
| # One or more tracking issues related to the change | ||
| issues: [21046] | ||
|
|
||
| # (Optional) One or more lines of additional information to render under the primary note. | ||
| # These lines will be padded with 2 spaces and then inserted directly into the document. | ||
| # Use pipe (|) for multiline entries. | ||
| subtext: |
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| include ../../Makefile.Common |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # Apache Spark Receiver | ||
|
|
||
| | Status | | | ||
| | ------------------------ | ---------------- | | ||
| | Stability | [development] | | ||
| | Supported pipeline types | metrics | | ||
| | Distributions | [contrib] | | ||
|
|
||
| This receiver fetches metrics for an Apache Spark cluster through the Apache Spark REST API - specifically, the /metrics/json, /api/v1/applications/[app-id]/stages, /api/v1/applications/[app-id]/executors, and /api/v1/applications/[app-id]/jobs endpoints. | ||
|
|
||
| ## Purpose | ||
|
|
||
| The purpose of this component is to allow monitoring of Apache Spark clusters and the applications running on them through the collection of performance metrics like memory utilization, CPU utilization, shuffle operations, garbage collection time, I/O operations, and more. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| This receiver supports Apache Spark versions: | ||
|
|
||
| - 3.3.2+ | ||
|
|
||
| ## Configuration | ||
|
|
||
| These configuration options are for connecting to an Apache Spark application. | ||
|
|
||
| The following settings are optional: | ||
|
|
||
| - `collection_interval`: (default = `60s`): This receiver collects metrics on an interval. This value must be a string readable by Golang's [time.ParseDuration](https://pkg.go.dev/time#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. | ||
| - `endpoint`: (default = `http://localhost:4040`): Apache Spark endpoint to connect to in the form of `[http][://]{host}[:{port}]` | ||
| - `whitelisted_application_names`: An array of Spark application names for which metrics should be collected. If no application names are specified, metrics will be collected for all Spark applications running on the cluster at the specified endpoint. | ||
|
|
||
| ### Example Configuration | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| apachespark: | ||
| collection_interval: 60s | ||
| endpoint: http://localhost:4040 | ||
| whitelisted_application_names: | ||
| - PythonStatusAPIDemo | ||
| - PythonLR | ||
| ``` | ||
| The full list of settings exposed for this receiver are documented [here](./config.go) with detailed sample configurations [here](./testdata/config.yaml). | ||
| ## Metrics | ||
| Details about the metrics produced by this receiver can be found in [metadata.yaml](./metadata.yaml) | ||
| [development]: https://github.com/open-telemetry/opentelemetry-collector#development | ||
| [contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package apachesparkreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachesparkreceiver" | ||
|
|
||
| import ( | ||
| "errors" | ||
| "net/url" | ||
| "time" | ||
|
|
||
| "go.opentelemetry.io/collector/config/confighttp" | ||
| "go.opentelemetry.io/collector/receiver/scraperhelper" | ||
| ) | ||
|
|
||
| const ( | ||
| defaultCollectionInterval = time.Minute | ||
| defaultEndpoint = "http://localhost:4040" | ||
| ) | ||
|
|
||
| var ( | ||
| errInvalidEndpoint = errors.New("'endpoint' must be in the form of <scheme>://<hostname>:<port>") | ||
| ) | ||
|
|
||
| // Config defines the configuration for the various elements of the receiver agent. | ||
| type Config struct { | ||
| scraperhelper.ScraperControllerSettings `mapstructure:",squash"` | ||
| confighttp.HTTPClientSettings `mapstructure:",squash"` | ||
| WhitelistedApplicationNames []string `mapstructure:"whitelisted_application_names"` | ||
| } | ||
|
|
||
| // Validate validates missing and invalid configuration fields. | ||
| func (cfg *Config) Validate() error { | ||
| _, parseErr := url.Parse(cfg.Endpoint) | ||
| if parseErr != nil { | ||
| return errInvalidEndpoint | ||
| } | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package apachesparkreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachesparkreceiver" | ||
|
|
||
| import ( | ||
| "testing" | ||
|
|
||
| "github.com/stretchr/testify/require" | ||
| "go.opentelemetry.io/collector/config/confighttp" | ||
| ) | ||
|
|
||
| func TestValidate(t *testing.T) { | ||
| testCases := []struct { | ||
| desc string | ||
| cfg *Config | ||
| expectedErr error | ||
| }{ | ||
| { | ||
| desc: "default config", | ||
| cfg: &Config{}, | ||
| expectedErr: nil, | ||
| }, | ||
| { | ||
| desc: "invalid endpoint", | ||
| cfg: &Config{ | ||
| HTTPClientSettings: confighttp.HTTPClientSettings{ | ||
| Endpoint: "invalid://endpoint 12efg", | ||
| }, | ||
| }, | ||
| expectedErr: errInvalidEndpoint, | ||
| }, | ||
| } | ||
|
|
||
| for _, tc := range testCases { | ||
| t.Run(tc.desc, func(t *testing.T) { | ||
| actualErr := tc.cfg.Validate() | ||
| if tc.expectedErr != nil { | ||
| require.EqualError(t, actualErr, tc.expectedErr.Error()) | ||
| } else { | ||
| require.NoError(t, actualErr) | ||
| } | ||
| }) | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package apachesparkreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/apachesparkreceiver" |
Oops, something went wrong.