GitHub and GitLab OnPremise #156

LaurentMarchelli · 2023-10-18T14:29:12Z

Hi guys,

Your work looks promising !!!
However, is it reserved for official GitHub and Gitlab instances or "On Premise" instances will be supported as well with the appropriate configuration ?
For instance :

config.yaml

---
extensions:
  bearertokenauth/github:
    token: "ghp_thisismytoken"
receivers:
  gitprovider:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp
    # https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md
    endpoint: "https://ghetest.mycompany.grp/api/graphql"
    tls:
      insecure: false
      insecure_skip_verify: true
    scrapers:
      github:
        github_org: "target_organization"
        auth:
          authenticator: "bearertokenauth/github"
        metrics:
          git.repository.branch.count:
            enabled: true
          git.repository.branch.time:
            enabled: true
          git.repository.contributor.count:
            enabled: true
          git.repository.count:
            enabled: true
          git.repository.pull_request.time:
            enabled: true
        resource_attributes:
          git.vendor.name:
            enabled: true
          organization.name:
            enabled: true
exporters:
  logging:
    verbosity: detailed
service:
  extensions: [bearertokenauth/github]
  pipelines:
    metrics:
      receivers: [gitprovider]
      exporters: [logging]

4lch4 · 2023-10-18T17:39:09Z

Thank you for the input!

This can definitely be done, I'll get it knocked out this week and submit a PR 😊
I'm really curious, how did you find our repo? We're still in early stages so it's really exciting to see other people submitting issues.

LaurentMarchelli · 2023-10-18T18:38:29Z

Hi Devin,

It was definitely an adventure, started on opentelemetry-collector-contrib to identify every kind of receiver that can be used to analyze CI troubles and produce a status dashboard.

We tried to use your Git Provider Receiver as defined inside the contrib repository but we had trouble to run it ...
Guess what ? It is not not embedded into the contrib docker image 0.87.0.
https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-contrib/manifest.yaml

From the chat open-telemetry/opentelemetry-collector-contrib#22028, we knew the development got a sponsor and work is in progress, so we tried the docker image from the upstream referenced here :
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/gitproviderreceiver/README.md

But because you are really in early development stages and as I understood you are working on refactoring, the last available image runnable on our k8s did not worked as expected, even with GitHub.com (ghcr.io/liatrio/liatrio-otel-collector:0.28.0-386).

So, we are looking forward for new version to try with our on premise GitHub Enterprise. ;-)

Best regards,
Laurent Marchelli

4lch4 · 2023-10-18T19:02:59Z

I cannot thank you enough for this input, Laurent! I'll bring this back to our team and we'll make sure to address these issues for you :)

adrielp · 2023-10-18T19:06:15Z

@LaurentMarchelli awesome, thank you for sharing! Yea, since it's in development, it won't be published in their release until we contribute more code & update it to alpha.

We have a demo of the collector in how to configure it in the https://github.com/liatrio/o11y-demo repo. It's basically the OTEL demo but focused on git metrics with some dashboards to reflect them. I'm in the OTEL slack workspace if you ever want to reach out there for faster conversation & huddle on it! Would be happy to chat anytime.

Can I ask what didn't work on your k8s deployment with regular git for the 0.28.0 version?

Also, I'm picking this task up right now to enable custom URLs through the default endpoint config. Should be done shortly.

LaurentMarchelli · 2023-10-18T19:49:28Z

Hi guys,

Thanks for the detailed information, I will definitely take a look at the demo and the latest configuration changes.
We look forward to setting up the endpoint and playing intensively with our on-premises instance.

As the version 0.29.0 is now published I will try it soon on GitHub.com with a token. (ghcr.io/liatrio/liatrio-otel-collector:0.29.0-386)
@adrielp, please find log lines below, I hope it will help.

2023-10-18T13:23:20.953Z info service/telemetry.go:84 Setting up own telemetry...
2023-10-18T13:23:20.954Z info service/telemetry.go:201 Serving Prometheus metrics {"address": ":8888", "level": "Basic"}
2023-10-18T13:23:20.954Z info [email protected]/exporter.go:275 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-10-18T13:23:20.954Z info [email protected]/receiver.go:296 Development component. May change in the future. {"kind": "receiver", "name": "gitprovider", "data_type": "metrics"}
2023-10-18T13:23:20.957Z info service/service.go:138 Starting otelcol-custom... {"Version": "1.0.0", "NumCPU": 6}
2023-10-18T13:23:20.957Z info extensions/extensions.go:31 Starting extensions...
2023-10-18T13:23:20.957Z info extensions/extensions.go:34 Extension is starting... {"kind": "extension", "name": "bearertokenauth/github"}
2023-10-18T13:23:20.957Z info extensions/extensions.go:38 Extension started. {"kind": "extension", "name": "bearertokenauth/github"}
2023-10-18T13:23:20.957Z info githubscraper/github_scraper.go:46 Starting the scraper inside scraper.go {"kind": "receiver", "name": "gitprovider", "data_type": "metrics"}
2023-10-18T13:23:20.957Z info service/service.go:161 Everything is ready. Begin running and processing data.
panic: runtime error: index out of range [2] with length 2
goroutine 102 [running]:
github.com/liatrio/liatrio-otel-collector/pkg/receiver/gitproviderreceiver/internal/scraper/githubscraper.(*githubScraper).scrape(0xf728b10, {0xbd70bc0, 0xf412318})
 /home/runner/work/liatrio-otel-collector/liatrio-otel-collector/pkg/receiver/gitproviderreceiver/internal/scraper/githubscraper/github_scraper.go:396 +0x1f55
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape(...)
 /home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport(0xf73a9a0)
 /home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:197 +0x137
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1()
 /home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:172 +0xec
created by go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping
 /home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/scraperhelper/scrapercontroller.go:158 +0x5a

adrielp · 2023-10-18T20:30:45Z

@LaurentMarchelli - just published v0.30.0 which has overridable endpoints. Let me know if that works.

The endpoint & tls values are actually defined in the scraper sections of the configs instead of the receiver.

So config should look like this:

receivers:
    gitprovider:
        initial_delay: 1s
        collection_interval: 60s
        scrapers:
            github:
                github_org: myfancyorg
                endpoint: "https://selfmanagedenterpriseserver.com"
                tls:
                    insecure: false
                    insecure_skip_verify: true
                auth: ....

LaurentMarchelli · 2023-10-19T10:14:50Z

Hi @adrielp,

It’s great, thanks to your latest changes, version 0.30.0 now works with version 3.8.1 of Gitlab Enterprise on-premises.

Guys, you rocks !

Just one point: during initial testing, it didn’t work because the program added the path /graphql to the endpoint and I’m not sure it’s best practice because:

The intended path is /api/graphql
The path can be modified according to the ghe versions.
The full URL is the one referenced in the ghe documentation
(Forming calls with GraphQL - GitHub Enterprise Server 3.8 Docs)
The full URL is the one used with the altair-graphql/altair explorer
(altair-graphql/altair: GraphQL Client for all platforms)

And I would have prefered to use :

        endpoint: "https://ghetest.mycompany.grp:443/api/graphql"

Instead than

        endpoint: "https://ghetest.mycompany.grp:443/api"

Here is my working configuration (anonymized):

config.yaml

---
extensions:
  bearertokenauth/github:
    token: "ghp_************************************"
receivers:
  gitprovider:
    # https://github.com/liatrio/liatrio-otel-collector/blob/main/pkg/receiver/gitproviderreceiver/README.md
    # https://github.com/liatrio/liatrio-otel-collector/blob/main/pkg/receiver/gitproviderreceiver/documentation.md
    # initial_delay: 1s
    # collection_interval: (default = 30s)
    initial_delay: 1s
    collection_interval: 60s
    scrapers:
      github:
        # https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md
        # https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp
        # https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md
        github_org: "target_organization"
        endpoint: "https://ghetest.mycompany.grp:443/api"
        tls:
          insecure: false
          insecure_skip_verify: true
        auth:
          authenticator: "bearertokenauth/github"
        metrics:
          git.repository.count:
            enabled: true
exporters:
  logging:
    verbosity: detailed
service:
  extensions: [bearertokenauth/github]
  pipelines:
    metrics:
      receivers: [gitprovider]
      exporters: [logging]

Thank you very much, we stay in touch for further feedbacks.
Best regards,

Laurent Marchelli

adrielp · 2023-10-19T14:03:04Z

@LaurentMarchelli thanks for trying it and the feedback! I can make a quick fix to auto append api/graphql. The reason i selected to append it instead of forcing the user to supply the whole URL is because this endpoint is also used for the REST API which has a different context path (it's just / at the end of the domain. REST is used for the contributor count call because GraphQL doesn't support that metric natively.

The receiver would have to make multiple calls per repo going back in time for the entire commit history of the main branch, create a unique list of users, and then count them to be able to do that through graphql.

LaurentMarchelli · 2023-10-19T16:21:55Z

@adrielp,

Thanks for the explanation, now it makes sense.
However, there is no need to spend time on a new version just for the quick fix, because git.repository.contributor.count is disabled in my configuration and I can handle the partial URL.

I will focus on metrics now.
Regards,

Laurent

adrielp · 2023-10-23T19:59:48Z

Ok, thanks! Going to go ahead and close this as done. Will open another piece of work to improve the endpoint handling & document the default reasoning.

LaurentMarchelli changed the title ~~GitHub and Gilab OnPremise~~ GitHub and GitLab OnPremise Oct 18, 2023

4lch4 self-assigned this Oct 18, 2023

4lch4 added this to the Increase o11y adoption internally and externally - v1 milestone Oct 18, 2023

4lch4 removed their assignment Oct 19, 2023

adrielp self-assigned this Oct 19, 2023

adrielp added enhancement New feature or request go labels Oct 19, 2023

adrielp closed this as completed Oct 23, 2023

adrielp mentioned this issue Oct 23, 2023

Improve endpoint handling for github and gitlab scrapers #165

Closed

LaurentMarchelli mentioned this issue Oct 27, 2023

fix: Endpoint management for GitHub enterprise (ghe) #176

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub and GitLab OnPremise #156

GitHub and GitLab OnPremise #156

LaurentMarchelli commented Oct 18, 2023

4lch4 commented Oct 18, 2023

LaurentMarchelli commented Oct 18, 2023 •

edited

Loading

4lch4 commented Oct 18, 2023

adrielp commented Oct 18, 2023

LaurentMarchelli commented Oct 18, 2023

adrielp commented Oct 18, 2023

LaurentMarchelli commented Oct 19, 2023

adrielp commented Oct 19, 2023

LaurentMarchelli commented Oct 19, 2023

adrielp commented Oct 23, 2023

GitHub and GitLab OnPremise #156

GitHub and GitLab OnPremise #156

Comments

LaurentMarchelli commented Oct 18, 2023

4lch4 commented Oct 18, 2023

LaurentMarchelli commented Oct 18, 2023 • edited Loading

4lch4 commented Oct 18, 2023

adrielp commented Oct 18, 2023

LaurentMarchelli commented Oct 18, 2023

adrielp commented Oct 18, 2023

LaurentMarchelli commented Oct 19, 2023

adrielp commented Oct 19, 2023

LaurentMarchelli commented Oct 19, 2023

adrielp commented Oct 23, 2023

LaurentMarchelli commented Oct 18, 2023 •

edited

Loading