[receiver/gitproviderreceiver] add branch time, commit, and line metrics

.chloggen/branch-metrics.yaml

@@ -0,0 +1,34 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: gitproviderreceiver
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Adds branch commit and line based metrics
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [22028]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext: |
+  Adds the following branch based metrics.
+  * git.repository.branch.time
+  * git.repository.branch.commit.aheadby.count
+  * git.repository.branch.commit.behindby.count
+  * git.repository.branch.line.deletion.count
+  * git.repository.branch.line.addition.count
+
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: []

receiver/gitproviderreceiver/README.md

@@ -19,35 +19,17 @@ The current default set of metrics common across all vendors can be found in [do
 
 These default metrics can be used as leading indicators to the DORA metrics; helping provide insight into modern-day engineering practices.
 
-## GitHub Metrics
-
-The current metrics available via scraping from GitHub are:
-
-- [x] Repository count
-- [ ] Repository branch time
-- [x] Repository branch count
-- [x] Repository contributor count
-- [x] Repository pull request open time
-- [x] Repository pull request time to merge
-- [ ] Repository pull request deployment time
-- [x] Repository pull request time to approval
-- [x] Repository pull request count | stores an attribute of `pull_request.state` equal to `open` or `merged`
-
-> Note: Some metrics may be disabled by default and have to be explicitly enabled.
-> For example, the repository contributor count metric is one such metric. This is
-> because this metric relies on the REST API which is subject to lower rate limits.
-
 ## Getting Started
 
 The collection interval is common to all scrapers and is set to 30 seconds by default.
 
 > Note: Generally speaking, if the vendor allows for anonymous API calls, then you
 > won't have to configure any authentication, but you may only see public repositories
-> and organizations.
+> and organizations. You may run into significantly more rate limiting.
 
 ```yaml
 gitprovider:
-    collection_interval: <duration> #default = 30s
+    collection_interval: <duration> #default = 30s recommended 300s
     scrapers:
         <scraper1>:
         <scraper2>:
@@ -71,7 +53,7 @@ receivers:
                     git.repository.contributor.count:
                         enabled: true
                 github_org: myfancyorg
-                search_query: "org:myfancyorg topic:o11yalltheway" #optional query override, defaults to "{org,user}:<github_org>"
+                search_query: "org:myfancyorg topic:o11yalltheway" #Recommended optional query override, defaults to "{org,user}:<github_org>"
                 endpoint: "https://selfmanagedenterpriseserver.com"
                 auth:
                     authenticator: bearertokenauth/github
@@ -87,7 +69,44 @@ service:
 This receiver is developed upstream in the [liatrio-otel-collector distribution](https://github.com/liatrio/liatrio-otel-collector)
 where a quick start exists with an [example config](https://github.com/liatrio/liatrio-otel-collector/blob/main/config/config.yaml)
 
+A Grafana Dashboard exists on the marketplace for this receiver and can be
+found [here](https://grafana.com/grafana/dashboards/20976-engineering-effectiveness-metrics/).
+
 The available scrapers are:
 | Scraper  | Description             |
 |----------|-------------------------|
 | [github] | Git Metrics from [GitHub](https://github.com/) |
+
+## GitHub Scraper
+
+> Important: 
+> * The GitHub scraper does not emit metrics for branches that have not had
+>   changes since creation from the default branch (trunk).
+> * Due to GitHub API limitations, it is possible for the branch time metric to
+>   change when rebases occur, recreating the commits with new timestamps.
+
+
+For additional context on GitHub scraper limitations and inner workings please
+see the [GitHub Scraper README][ghsread].
+
+[ghsread]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/gitproviderreceiver/internal/scraper/githubscraper/README.md#github-limitations
+
+
+The current metrics available via scraping from GitHub are:
+
+- [x] Repository count
+- [x] Repository contributor count
+- [x] Repository branch count
+- [x] Repository branch time
+- [x] Repository branch commit aheadby count
+- [x] Repository branch commit behindby count
+- [x] Repository branch line addition count
+- [x] Repository branch line deletion count
+- [x] Repository pull request open time
+- [x] Repository pull request time to merge
+- [x] Repository pull request time to approval
+- [x] Repository pull request count | stores an attribute of `pull_request.state` equal to `open` or `merged`
+
+> Note: Some metrics may be disabled by default and have to be explicitly enabled.
+> For example, the repository contributor count metric is one such metric. This is
+> because this metric relies on the REST API which is subject to lower rate limits.

receiver/gitproviderreceiver/documentation.md

@@ -12,9 +12,39 @@ metrics:
     enabled: false
 ```
 
+### git.repository.branch.commit.aheadby.count
+
+The number of commits a branch is ahead of the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {commit} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
+### git.repository.branch.commit.behindby.count
+
+The number of commits a branch is behind the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {commit} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
 ### git.repository.branch.count
 
-Number of branches in a repository
+The number of branches in a repository.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -26,17 +56,62 @@ Number of branches in a repository
 | ---- | ----------- | ------ |
 | repository.name | The name of a Git repository | Any Str |
 
+### git.repository.branch.line.addition.count
+
+The number of lines added in a branch relative to the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {line} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
+### git.repository.branch.line.deletion.count
+
+The number of lines deleted in a branch relative to the default branch (trunk).
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| {line} | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
+### git.repository.branch.time
+
+Time a branch created from the default branch (trunk) has existed.
+
+| Unit | Metric Type | Value Type |
+| ---- | ----------- | ---------- |
+| s | Gauge | Int |
+
+#### Attributes
+
+| Name | Description | Values |
+| ---- | ----------- | ------ |
+| repository.name | The name of a Git repository | Any Str |
+| branch.name | The name of a Git branch | Any Str |
+
 ### git.repository.count
 
-Number of repositories in an organization
+The number of repositories in an organization.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
 | {repository} | Gauge | Int |
 
 ### git.repository.pull_request.count
 
-The number of pull requests in a repository, categorized by their state (either open or merged)
+The number of pull requests in a repository, categorized by their state (either open or merged).
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -51,7 +126,7 @@ The number of pull requests in a repository, categorized by their state (either
 
 ### git.repository.pull_request.time_open
 
-The amount of time a pull request has been open
+The amount of time a pull request has been open.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -66,7 +141,7 @@ The amount of time a pull request has been open
 
 ### git.repository.pull_request.time_to_approval
 
-The amount of time it took a pull request to go from open to approved
+The amount of time it took a pull request to go from open to approved.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -81,7 +156,7 @@ The amount of time it took a pull request to go from open to approved
 
 ### git.repository.pull_request.time_to_merge
 
-The amount of time it took a pull request to go from open to merged
+The amount of time it took a pull request to go from open to merged.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |
@@ -106,7 +181,7 @@ metrics:
 
 ### git.repository.contributor.count
 
-Total number of unique contributors to a repository
+The number of unique contributors to a repository.
 
 | Unit | Metric Type | Value Type |
 | ---- | ----------- | ---------- |

receiver/gitproviderreceiver/go.mod

@@ -6,6 +6,8 @@ require (
 	github.com/Khan/genqlient v0.7.0
 	github.com/google/go-cmp v0.6.0
 	github.com/google/go-github/v62 v62.0.0
+	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden v0.102.0
+	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest v0.0.0-00010101000000-000000000000
 	github.com/stretchr/testify v1.9.0
 	go.opentelemetry.io/collector/component v0.102.0
 	go.opentelemetry.io/collector/config/confighttp v0.102.0
@@ -29,10 +31,10 @@ require (
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
 	github.com/fsnotify/fsnotify v1.7.0 // indirect
-	github.com/go-logr/logr v1.4.1 // indirect
+	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
-	github.com/go-ole/go-ole v1.2.6 // indirect
-	github.com/go-viper/mapstructure/v2 v2.0.0-alpha.1 // indirect
+	github.com/go-ole/go-ole v1.3.0 // indirect
+	github.com/go-viper/mapstructure/v2 v2.0.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/snappy v0.0.4 // indirect
@@ -46,25 +48,26 @@ require (
 	github.com/knadh/koanf/maps v0.1.1 // indirect
 	github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
 	github.com/knadh/koanf/v2 v2.1.1 // indirect
-	github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
+	github.com/lufia/plan9stats v0.0.0-20240408141607-282e7b5d6b74 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil v0.102.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
 	github.com/prometheus/client_golang v1.19.1 // indirect
 	github.com/prometheus/client_model v0.6.1 // indirect
 	github.com/prometheus/common v0.53.0 // indirect
 	github.com/prometheus/procfs v0.15.0 // indirect
-	github.com/rs/cors v1.10.1 // indirect
+	github.com/rs/cors v1.11.0 // indirect
 	github.com/shirou/gopsutil/v3 v3.24.4 // indirect
 	github.com/shoenig/go-m1cpu v0.1.6 // indirect
 	github.com/spf13/cobra v1.8.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	github.com/tklauser/go-sysconf v0.3.12 // indirect
-	github.com/tklauser/numcpus v0.6.1 // indirect
-	github.com/vektah/gqlparser/v2 v2.5.11 // indirect
+	github.com/tklauser/go-sysconf v0.3.13 // indirect
+	github.com/tklauser/numcpus v0.7.0 // indirect
+	github.com/vektah/gqlparser/v2 v2.5.12 // indirect
 	github.com/yusufpapurcu/wmi v1.2.4 // indirect
 	go.opencensus.io v0.24.0 // indirect
 	go.opentelemetry.io/collector v0.102.0 // indirect
@@ -107,12 +110,18 @@ require (
 	go.uber.org/multierr v1.11.0 // indirect
 	golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect
 	golang.org/x/net v0.25.0 // indirect
-	golang.org/x/sys v0.20.0 // indirect
-	golang.org/x/text v0.15.0 // indirect
+	golang.org/x/sys v0.21.0 // indirect
+	golang.org/x/text v0.16.0 // indirect
 	gonum.org/v1/gonum v0.15.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20240520151616-dc85e6b867a5 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20240520151616-dc85e6b867a5 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240528184218-531527333157 // indirect
 	google.golang.org/grpc v1.64.0 // indirect
 	google.golang.org/protobuf v1.34.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil => ../../pkg/pdatautil
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest => ../../pkg/pdatatest
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/golden => ../../pkg/golden