Skip to content
This repository has been archived by the owner on Aug 13, 2019. It is now read-only.

Add statsd metrics #378

Closed
peterbe opened this issue Mar 21, 2018 · 13 comments
Closed

Add statsd metrics #378

peterbe opened this issue Mar 21, 2018 · 13 comments
Assignees

Comments

@peterbe
Copy link
Contributor

peterbe commented Mar 21, 2018

A lot is going on in the jobs code (Lambda handler and inventory-to-records) and we have very limited insight. Let's add metrics of counters and timers to see what's going and what's taking time.

To accomplish this we also need to file a bugzilla bug to Cloud Services to add the Datadog daemon on the server that runs the cron job.

@peterbe peterbe self-assigned this Mar 26, 2018
@peterbe
Copy link
Contributor Author

peterbe commented Mar 28, 2018

<wezhou> peterbe: as for statsd, you can send it to 127.0.0.1:8125
<peterbe> wezhou: Is that a datadog daemon?
<wezhou> peterbe: yes. it hasn't been installed on the instance where
the cron is running, so feel free to file a ticket to have
it installed.

note-to-self: not done yet.

@peterbe peterbe closed this as completed in abe3c58 Apr 5, 2018
@peterbe
Copy link
Contributor Author

peterbe commented Apr 12, 2018

Let's keep this open till it's been tested in stage
https://bugzilla.mozilla.org/show_bug.cgi?id=1453679

@peterbe peterbe reopened this Apr 12, 2018
@bqbn
Copy link
Collaborator

bqbn commented Apr 19, 2018

datadog agent is running on the admin instance.

I think you might need to cut a release and have it deployed to -stage for testing. Currently -stage and -prod are running the latest version, which is v1.1.5.

@peterbe
Copy link
Contributor Author

peterbe commented Apr 20, 2018

Ok. So once it's upgraded it'll work. I'll start on making a release.

By the way, @bqbn What does it take to make stage auto-deploy? I guess that's two questions in one:

  1. How can we upgrade stage based on master instead of manual tagged git shas?
  2. How can we upgrade the lambda function from automation rather from manual dev-make-lambda.zip-send-to-ops?

@bqbn
Copy link
Collaborator

bqbn commented Apr 20, 2018

How can we upgrade stage based on master instead of manual tagged git shas?

This is technically doable, but the convention is not to do that. We only install tagged version to -stage, not master.

Do you have a reason for which you must install "master" onto -stage?

How can we upgrade the lambda function from automation rather from manual dev-make-lambda.zip-send-to-ops?

Right now the only way to trigger a Jenkins pipeline from outside of Jenkins is via Dockerhub, as documented in https://github.com/mozilla-services/cloudops-deployment#code-flow

Basically we'd have to build a docker image and push that to docker hub, which then calls the jenkins-proxy to trigger a jenkins pipeline. And then the pipeline will need to download the image from docker hub, copy the lambda zip file out from the docker image, then deploy it.

Also note that, the lambda still needs to be tagged with a version, so that we can deploy it to -stage.

@peterbe
Copy link
Contributor Author

peterbe commented Apr 20, 2018

Ok. I think I spoke too soon. Stage should probably continue to be by tag and dockerhub. What would be ideal is that stage is automatically upgraded (without any ops/human intervention) when a new tag is pushed (and circle builds based on this successfully).

  • I can write a script to automate away the making of tags for devs. (e.g. make-tag.py)

  • What would it take to get the lambda.zip into AWS without having to manually make it locally and upload it to Bugzilla as an attachment?

  • Some magic on your end so that when a new tag lands successfully on CircleCI, it upgrades Stage automatically. (This is how Tecken works. Uses Circle and Docker etc. Built by miles and me)

@bqbn
Copy link
Collaborator

bqbn commented Apr 20, 2018

What would be ideal is that stage is automatically upgraded (without any ops/human intervention) when a new tag is pushed (and circle builds based on this successfully).

This is already the case.

What would it take to get the lambda.zip into AWS without having to manually make it locally and upload it to Bugzilla as an attachment?

I'm under the impression that you don't need to make it locally, rather, when you release a version in github, github provides you with an option to generate a zip file, you just need to click the button.

You don't need to upload it to Bugzilla, rather, you file a ticket to request deploying it to -stage with a version number, and our pipeline downloads it from github automatically.

One example location our pipeline comes to download is https://github.com/mozilla-services/buildhub/releases/download/v1.1.5/buildhub-lambda-v1.1.5.zip where "v1.1.5" is the version number. Please see https://github.com/mozilla-services/buildhub/releases.

This is how Tecken works.

If Tecken is a lambda, I'm happy to look at some examples of it and see if it fits in our current setup. Please share some links to it.

@peterbe
Copy link
Contributor Author

peterbe commented Apr 23, 2018

So, I've made a release. https://github.com/mozilla-services/buildhub/releases/tag/v1.2.0
It's got a buildhub-lambda-v1.2.0.zip asset.
A new GitHub Release plus a new lambda ZIP, will that upgrade Stage automatically now?

I wrote this script which does:

  1. Update the version string and the CHANGELOG.rst file.
  2. Makes a commit and a tag (naming and change log is all automated)
  3. Creates a Release
  4. Uploads a .zip file to the Release.

So, if I understand this, this'll upgrade Stage (and Stage's Lambda). Right?

@bqbn
Copy link
Collaborator

bqbn commented Apr 23, 2018

A new GitHub Release plus a new lambda ZIP, will that upgrade Stage automatically now?

Yes for the cron job. As a matter of fact, I see that our pipeline was triggered and the v1.2.0 version of cron was deployed to -stage.

No for the lambda. As mentioned in #378 (comment), unless the lambda is built into a docker image by circleci, there is no way to trigger our pipeline to deploy it.

So currently to deploy the lambda, the process is: file a ticket in bugzilla with the desired version number to have it deploy to -stage, then another ticket to deploy it to -prod.

@peterbe
Copy link
Contributor Author

peterbe commented Apr 23, 2018

Ah! Got it. I misunderstood the distinction between the cron job thing and the lambda. I'll amend my in-progress script to make the bugzilla filing easier.

@peterbe
Copy link
Contributor Author

peterbe commented Apr 24, 2018

@bqbn
Buildhub Stage has been upgraded (Lambda and cron job).
I'm pretty sure it's doing something since my recent fixes in Buildhub v1.2.0 is raising exceptions in Sentry for Python code that can only exist in Buildhub Stage.

The default config is that it sends statsd pings to localhost:8125.

@peterbe
Copy link
Contributor Author

peterbe commented Apr 24, 2018

The problem is that I can't create a Buildhub Performance dashboard and find any of my keys. They should look something like buildhub.s3_event_event.

This is the dashboard I started: https://app.datadoghq.com/dash/794559/buildhub-performance?live=true&page=0&is_auto=false&from_ts=1524596723306&to_ts=1524600323306&tile_size=m

@peterbe
Copy link
Contributor Author

peterbe commented Nov 19, 2018

We have statsd metrics for Buildhub now. Have had for a long time.

@peterbe peterbe closed this as completed Nov 19, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants