Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cmd history-exporter #8

Merged
merged 21 commits into from
Oct 22, 2024
Merged

Add cmd history-exporter #8

merged 21 commits into from
Oct 22, 2024

Conversation

fedordikarev
Copy link
Collaborator

  • Rearrange modules and make libraries reuasable
  • We want to have historical data analysis as well, for that create history-exporter that will import historical data
  • Disable docker push in ci for now, reenable it later

Copy link

@orca-security-us orca-security-us bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@fedordikarev
Copy link
Collaborator Author

omg, what I did here:

  1. I list workflows runs for a given period of time
  2. from that list I get id and attempt number
  3. if information for that id and attempt is already in DB, I skip it
  4. if not, I query gh api one more time for the info and save it into DB.

you also see it?
I already have all the information needed for the step 4 in results of step 1 (not sure yet, have to check it).
And if it's true, I could not only speed up that process, but also stop hitting rate limit for GH API, so "month long export" could be done in matter of tens of minutes, not tens of hours as it is right now.

Copy link
Member

@bayandin bayandin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave a brief look, so if it works — it works :)

Thanks!

.github/workflows/ci.yml Outdated Show resolved Hide resolved
pkg/export/export.go Outdated Show resolved Hide resolved
cmd/history-exporter/main.go Show resolved Hide resolved
fedordikarev and others added 5 commits October 22, 2024 06:24
* CI: push docker images to neondatabase org
* docker://yatheo -> docker://neondatabase
* github.com/fedordikarev ->  github.com/neondatabase
@fedordikarev fedordikarev merged commit e70d2ba into main Oct 22, 2024
4 checks passed
@fedordikarev fedordikarev deleted the feat/history_exporter branch October 22, 2024 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants