[receiver/gitproviderreceiver] add branch time, commit, and line metr…

…ics (#32812) **Description:** Adds the following branch based metrics. * git.repository.branch.time * git.repository.branch.commit.aheadby.count * git.repository.branch.commit.behindby.count * git.repository.branch.line.deletion.count * git.repository.branch.line.addition.count Additionally brings this receiver very close to the source version in the Liatrio Distro. Brings the testing up to par as well. After this pull request is merged, I'd like to move this component to alpha status so it can be built within the contrib binary and used by others. Once in alpha, I'd like to bring over the newest metric (repository.cve.count), bring over the GitLab scraper, and potentially propose a new component I developed to work in-tandem with this receiver, the GitHub App Authentication extension. **Link to tracking Issue:** #22028 **Testing:** Brings testing inline with source and include scrape testing. **Documentation:** Updated the documentation to reflect the additional metrics. --------- Co-authored-by: Curtis Robert <[email protected]>
open-telemetry · Jun 5, 2024 · 9e4a906 · 9e4a906
1 parent e7cf560
commit 9e4a906
Show file tree

Hide file tree

Showing 20 changed files with 2,088 additions and 111 deletions.
diff --git a/.chloggen/branch-metrics.yaml b/.chloggen/branch-metrics.yaml
@@ -0,0 +1,34 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: gitproviderreceiver
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Adds branch commit and line based metrics
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [22028]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext: |
+  Adds the following branch based metrics.
+  * git.repository.branch.time
+  * git.repository.branch.commit.aheadby.count
+  * git.repository.branch.commit.behindby.count
+  * git.repository.branch.line.deletion.count
+  * git.repository.branch.line.addition.count
+  
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: []
diff --git a/receiver/gitproviderreceiver/README.md b/receiver/gitproviderreceiver/README.md
@@ -19,35 +19,17 @@ The current default set of metrics common across all vendors can be found in [do
 
 These default metrics can be used as leading indicators to the DORA metrics; helping provide insight into modern-day engineering practices.
 
-## GitHub Metrics
-
-The current metrics available via scraping from GitHub are:
-
-- [x] Repository count
-- [ ] Repository branch time
-- [x] Repository branch count
-- [x] Repository contributor count
-- [x] Repository pull request open time
-- [x] Repository pull request time to merge
-- [ ] Repository pull request deployment time
-- [x] Repository pull request time to approval
-- [x] Repository pull request count | stores an attribute of `pull_request.state` equal to `open` or `merged`
-
-> Note: Some metrics may be disabled by default and have to be explicitly enabled.
-> For example, the repository contributor count metric is one such metric. This is
-> because this metric relies on the REST API which is subject to lower rate limits.
-
 ## Getting Started
 
 The collection interval is common to all scrapers and is set to 30 seconds by default.
 
 > Note: Generally speaking, if the vendor allows for anonymous API calls, then you
 > won't have to configure any authentication, but you may only see public repositories
-> and organizations.
+> and organizations. You may run into significantly more rate limiting.
 
 ```yaml
 gitprovider:
-    collection_interval: <duration> #default = 30s
+    collection_interval: <duration> #default = 30s recommended 300s
     scrapers:
         <scraper1>:
         <scraper2>:
@@ -71,7 +53,7 @@ receivers:
                     git.repository.contributor.count:
                         enabled: true
                 github_org: myfancyorg
-                search_query: "org:myfancyorg topic:o11yalltheway" #optional query override, defaults to "{org,user}:<github_org>"
+                search_query: "org:myfancyorg topic:o11yalltheway" #Recommended optional query override, defaults to "{org,user}:<github_org>"
                 endpoint: "https://selfmanagedenterpriseserver.com"
                 auth:
                     authenticator: bearertokenauth/github
@@ -87,7 +69,44 @@ service:
 This receiver is developed upstream in the [liatrio-otel-collector distribution](https://github.com/liatrio/liatrio-otel-collector)
 where a quick start exists with an [example config](https://github.com/liatrio/liatrio-otel-collector/blob/main/config/config.yaml)
 
+A Grafana Dashboard exists on the marketplace for this receiver and can be
+found [here](https://grafana.com/grafana/dashboards/20976-engineering-effectiveness-metrics/).
+
 The available scrapers are:
 | Scraper  | Description             |
 |----------|-------------------------|
 | [github] | Git Metrics from [GitHub](https://github.com/) |
+
+## GitHub Scraper
+
+> Important: 
+> * The GitHub scraper does not emit metrics for branches that have not had
+>   changes since creation from the default branch (trunk).
+> * Due to GitHub API limitations, it is possible for the branch time metric to
+>   change when rebases occur, recreating the commits with new timestamps.
+
+
+For additional context on GitHub scraper limitations and inner workings please
+see the [GitHub Scraper README][ghsread].
+
+[ghsread]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/gitproviderreceiver/internal/scraper/githubscraper/README.md#github-limitations
+
+
+The current metrics available via scraping from GitHub are:
+
+- [x] Repository count
+- [x] Repository contributor count
+- [x] Repository branch count
+- [x] Repository branch time
+- [x] Repository branch commit aheadby count
+- [x] Repository branch commit behindby count
+- [x] Repository branch line addition count
+- [x] Repository branch line deletion count
+- [x] Repository pull request open time
+- [x] Repository pull request time to merge
+- [x] Repository pull request time to approval
+- [x] Repository pull request count | stores an attribute of `pull_request.state` equal to `open` or `merged`
+
+> Note: Some metrics may be disabled by default and have to be explicitly enabled.
+> For example, the repository contributor count metric is one such metric. This is
+> because this metric relies on the REST API which is subject to lower rate limits.
diff --git a/receiver/gitproviderreceiver/documentation.md b/receiver/gitproviderreceiver/documentation.md
@@ -12,9 +12,39 @@ metrics:
     enabled: false
 ```
 
+### git.repository.branch.commit.aheadby.count
+
+The number of commits a branch is ahead of the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {commit} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
+### git.repository.branch.commit.behindby.count
+
+The number of commits a branch is behind the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {commit} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
 ### git.repository.branch.count
 
-Number of branches in a repository
+The number of branches in a repository.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -26,17 +56,62 @@ Number of branches in a repository
 | ---- | ----------- | ------ |
 | repository.name | The name of a Git repository | Any Str |
 
+### git.repository.branch.line.addition.count
+
+The number of lines added in a branch relative to the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {line} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
+### git.repository.branch.line.deletion.count
+
+The number of lines deleted in a branch relative to the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {line} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
+### git.repository.branch.time
+
+Time a branch created from the default branch (trunk) has existed.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| s | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
 ### git.repository.count
 
-Number of repositories in an organization
+The number of repositories in an organization.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
 | {repository} | Gauge | Int |
 
 ### git.repository.pull_request.count
 
-The number of pull requests in a repository, categorized by their state (either open or merged)
+The number of pull requests in a repository, categorized by their state (either open or merged).
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -51,7 +126,7 @@ The number of pull requests in a repository, categorized by their state (either
 
 ### git.repository.pull_request.time_open
 
-The amount of time a pull request has been open
+The amount of time a pull request has been open.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -66,7 +141,7 @@ The amount of time a pull request has been open
 
 ### git.repository.pull_request.time_to_approval
 
-The amount of time it took a pull request to go from open to approved
+The amount of time it took a pull request to go from open to approved.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -81,7 +156,7 @@ The amount of time it took a pull request to go from open to approved
 
 ### git.repository.pull_request.time_to_merge
 
-The amount of time it took a pull request to go from open to merged
+The amount of time it took a pull request to go from open to merged.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -106,7 +181,7 @@ metrics:
 
 ### git.repository.contributor.count
 
-Total number of unique contributors to a repository
+The number of unique contributors to a repository.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |

diff --git a/receiver/gitproviderreceiver/go.mod b/receiver/gitproviderreceiver/go.mod
@@ -6,6 +6,8 @@ require (
 	github.com/Khan/genqlient v0.7.0
 	github.com/google/go-cmp v0.6.0
 	github.com/google/go-github/v62 v62.0.0
+	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden v0.102.0
+	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest v0.0.0-00010101000000-000000000000
 	github.com/stretchr/testify v1.9.0
 	go.opentelemetry.io/collector/component v0.102.0
 	go.opentelemetry.io/collector/config/confighttp v0.102.0
@@ -29,10 +31,10 @@ require (
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
 	github.com/fsnotify/fsnotify v1.7.0 // indirect
-	github.com/go-logr/logr v1.4.1 // indirect
+	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
-	github.com/go-ole/go-ole v1.2.6 // indirect
-	github.com/go-viper/mapstructure/v2 v2.0.0-alpha.1 // indirect
+	github.com/go-ole/go-ole v1.3.0 // indirect
+	github.com/go-viper/mapstructure/v2 v2.0.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/snappy v0.0.4 // indirect
@@ -46,25 +48,26 @@ require (
 	github.com/knadh/koanf/maps v0.1.1 // indirect
 	github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
 	github.com/knadh/koanf/v2 v2.1.1 // indirect
-	github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
+	github.com/lufia/plan9stats v0.0.0-20240408141607-282e7b5d6b74 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil v0.102.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
 	github.com/prometheus/client_golang v1.19.1 // indirect
 	github.com/prometheus/client_model v0.6.1 // indirect
 	github.com/prometheus/common v0.53.0 // indirect
 	github.com/prometheus/procfs v0.15.0 // indirect
-	github.com/rs/cors v1.10.1 // indirect
+	github.com/rs/cors v1.11.0 // indirect
 	github.com/shirou/gopsutil/v3 v3.24.4 // indirect
 	github.com/shoenig/go-m1cpu v0.1.6 // indirect
 	github.com/spf13/cobra v1.8.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	github.com/tklauser/go-sysconf v0.3.12 // indirect
-	github.com/tklauser/numcpus v0.6.1 // indirect
-	github.com/vektah/gqlparser/v2 v2.5.11 // indirect
+	github.com/tklauser/go-sysconf v0.3.13 // indirect
+	github.com/tklauser/numcpus v0.7.0 // indirect
+	github.com/vektah/gqlparser/v2 v2.5.12 // indirect
 	github.com/yusufpapurcu/wmi v1.2.4 // indirect
 	go.opencensus.io v0.24.0 // indirect
 	go.opentelemetry.io/collector v0.102.0 // indirect
@@ -107,12 +110,18 @@ require (
 	go.uber.org/multierr v1.11.0 // indirect
 	golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect
 	golang.org/x/net v0.25.0 // indirect
-	golang.org/x/sys v0.20.0 // indirect
-	golang.org/x/text v0.15.0 // indirect
+	golang.org/x/sys v0.21.0 // indirect
+	golang.org/x/text v0.16.0 // indirect
 	gonum.org/v1/gonum v0.15.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20240520151616-dc85e6b867a5 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20240520151616-dc85e6b867a5 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240528184218-531527333157 // indirect
 	google.golang.org/grpc v1.64.0 // indirect
 	google.golang.org/protobuf v1.34.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil => ../../pkg/pdatautil
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest => ../../pkg/pdatatest
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden => ../../pkg/golden