[Issue #2481] Add new GitHub export #2539
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Enables users to run the slack integration tests if they pass the: `--slack-token-set` flag
This dataset stores a newly transformed version of exported GitHub data.
Adds tests for get_parent_with_type()
Enables us to store multiple scripts in the integrations/github/ dir
- export_sprint_data() - export_roadmap_data()
Catches and logs validation errors instead of panicking during `populate_issue_lookup()`
Replaces numpy.nan with None in BaseDataset.to_dict() method
Exports sprint and roadmap data using the new structure
|
Approach looks great so far 🚀 |
Exports data required to build the new `GitHubIssues` dataset
Loads a dataset directly from a JSON file with the same shape.
Loads the `GitHubIssues` dataset instead of `SprintBoard` into the db. The `SprintBoard` represents the old method for exporting gh data, and `GitHubIssues` is the new way GitHub data is represented
|
👀 |
coilysiren
reviewed
Oct 28, 2024
| @@ -163,6 +164,17 @@ roadmap-data-export: | |||
| --project $(ROADMAP_PROJECT) \ | |||
| --output-file $(ROADMAP_FILE) | |||
|
|
|||
| delivery-data-export: | |||
There was a problem hiding this comment.
Do we want this running in AWS?
There was a problem hiding this comment.
Not yet! I'll think want to finish some subset of the following tickets:
- Turn on Sprinty McBurndown for sprint burndown
- Post new % completed by deliverable metric to Slack
- Remove old deliverable reports from
/analyticscodebase
Then I'll create a separate ticket for which commands to run in AWS and tag you on it @coilysiren
coilysiren
previously approved these changes
Oct 28, 2024
|
Looks great once we are green. Noting the documentation isn't updated but that could wait for a follow-up. There is this analytics doc ticket that could be used #2130 or another created. |
The extra spaces in the SPRINT_FIELD and POINTS_FIELD before the comment were being passed through to the CLI
|
@coilysiren and @DavidDudas-Intuitial I fixed one bug with the Makefile (fix(analytics): Extra space in Makefile variable) that I didn't catch until I was working on the sprint burndown PR, would you all mind reviewing again? |
acouch
approved these changes
Oct 29, 2024
coilysiren
approved these changes
Oct 29, 2024
2 tasks
DavidDudas-Intuitial
approved these changes
Oct 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds new export command for the new GitHub issue structure.
Fixes #2481
Time to review: 10 mins
Changes proposed
GitHubIssuesdataset todatasets/issues.py--slack-token-setpytest flagGitHubIssuesdataset that stores the content of the new delivery data exportintegrations/github.pyto make it a sub-packageintegrations/github/to store the bash script and graphql queriesexport_delivery_data()tointegrations/github/main.pyexport_json_to_database()withGitHubIssuesTODO
integrations/github.pyto make it a sub-packageintegrations/github/to store the bash script and graphql queriesexport_delivery_data()tointegrations/github/main.pyexport_json_to_database()Context for reviewers
documentation/analytics/development.mdusing Docker.make delivery-data-export POINTS_FIELD="Story Points"to export GitHub data using the new formatmake gh-data-db-importto load the exported data into the Postgres DBAdditional information