Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weekly report script #50

Merged
merged 1 commit into from
Aug 27, 2021
Merged

Weekly report script #50

merged 1 commit into from
Aug 27, 2021

Conversation

zackkrida
Copy link
Member

@zackkrida zackkrida commented Aug 14, 2021

Maybe I'll clean this up, maybe I won't 😆 written in a silly fp-lite style that I tend to write my own stuff in. Here's a sample output:

> node js/weekly_report.js 2021-08-09 | pbcopy
## [openverse](https://github.com/wordpress/openverse) 

- [#49 [Bug] Debug failing sync action](https://github.com/WordPress/openverse/issues/49)
- [#48 Include CODEOWNERS file in repo sync](https://github.com/WordPress/openverse/pull/48)
- [#47 Add an "internal improvement" label](https://github.com/WordPress/openverse/pull/47)
- [#46 Add banner asset with BG color to be independent of GitHub theme](https://github.com/WordPress/openverse/pull/46)
- [#45 Add the logo to the readme 😄 ](https://github.com/WordPress/openverse/pull/45)
- [#44 Add icon, logo and brand usage guidelines](https://github.com/WordPress/openverse/pull/44)
- [#43 Automation improvements](https://github.com/WordPress/openverse/pull/43)
- [#41 Move Openverse Catalog docs from GitHub to Make Handbook](https://github.com/WordPress/openverse/issues/41)
- [#15 Exclude draft PRs when adding new PRs to project board](https://github.com/WordPress/openverse/issues/15)

## [openverse-frontend](https://github.com/wordpress/openverse-frontend) 

- [#126 Translate default `description` meta tag](https://github.com/WordPress/openverse-frontend/pull/126)
- [#124 Speed up pre-commit hooks](https://github.com/WordPress/openverse-frontend/pull/124)
- [#122 [Feature] Make pre-commit hooks faster](https://github.com/WordPress/openverse-frontend/issues/122)
- [#120 Add correct meta video filters to the new filter method](https://github.com/WordPress/openverse-frontend/pull/120)
- [#115 Improve i18n pluralization for Russian](https://github.com/WordPress/openverse-frontend/pull/115)
- [#114 🔄 Synced file(s) with WordPress/openverse](https://github.com/WordPress/openverse-frontend/pull/114)
- [#111 Add AudioService and MediaProviderService](https://github.com/WordPress/openverse-frontend/pull/111)
- [#103 [Audio] Create Audio Service and Audio Provider Service](https://github.com/WordPress/openverse-frontend/issues/103)
- [#99 Replace Bulma helpers with Tailwind utilities](https://github.com/WordPress/openverse-frontend/pull/99)
- [#97 [Feature] Replace utility classes with Tailwind](https://github.com/WordPress/openverse-frontend/issues/97)
- [#42 Provide HTML code to allow CC Search to be embedded in other websites (original #505)](https://github.com/WordPress/openverse-frontend/issues/42)
- [#35 Allow users to submit CC-licensed content to Search (original #441)](https://github.com/WordPress/openverse-frontend/issues/35)

## [openverse-api](https://github.com/wordpress/openverse-api) 

- [#173 Add uppercase extension for case-sensitive file systems](https://github.com/WordPress/openverse-api/pull/173)
- [#170 Add the logo and make small text fixes in the API docs](https://github.com/WordPress/openverse-api/pull/170)
- [#168 Update content providers to match the new sample data](https://github.com/WordPress/openverse-api/pull/168)
- [#167 Bump boto3 from 1.18.11 to 1.18.16 in /openverse-api](https://github.com/WordPress/openverse-api/pull/167)
- [#166 Bump gevent from 21.1.2 to 21.8.0 in /openverse-api](https://github.com/WordPress/openverse-api/pull/166)
- [#165 Bump django from 3.2.5 to 3.2.6 in /openverse-api](https://github.com/WordPress/openverse-api/pull/165)
- [#164 🔄 Synced file(s) with WordPress/openverse](https://github.com/WordPress/openverse-api/pull/164)
- [#163 🔄 Synced file(s) with WordPress/openverse](https://github.com/WordPress/openverse-api/pull/163)
- [#160 Define integration tests for audio (and general improvements)](https://github.com/WordPress/openverse-api/pull/160)
- [#156 [Feature] Add image width and height to the API response](https://github.com/WordPress/openverse-api/issues/156)
- [#139 [Bug] [ingestion-server] Tldextract caching error](https://github.com/WordPress/openverse-api/issues/139)
- [#108 Add DRF endpoints for audio](https://github.com/WordPress/openverse-api/pull/108)
- [#74 Create API endpoints for audio](https://github.com/WordPress/openverse-api/issues/74)

## [openverse-catalog](https://github.com/wordpress/openverse-catalog) 

- [#137 Replace `genre` property with `genres` in tests](https://github.com/WordPress/openverse-catalog/pull/137)
- [#136 Add example vars for airflow remote logging](https://github.com/WordPress/openverse-catalog/pull/136)
- [#135 Delete docs folder](https://github.com/WordPress/openverse-catalog/pull/135)
- [#134 🔄 Synced file(s) with WordPress/openverse](https://github.com/WordPress/openverse-catalog/pull/134)
- [#131 Add a PR template to the repository](https://github.com/WordPress/openverse-catalog/pull/131)
- [#130 Modify audio columns](https://github.com/WordPress/openverse-catalog/pull/130)
- [#129 Make wikimedia script pass license_info, not license_url](https://github.com/WordPress/openverse-catalog/pull/129)
- [#128 Add a script to create provider API script template](https://github.com/WordPress/openverse-catalog/pull/128)
- [#126 Delete duplicated CommonCrawl providers](https://github.com/WordPress/openverse-catalog/pull/126)
- [#125 Add missing `watermarked` column to audio loading table](https://github.com/WordPress/openverse-catalog/pull/125)
- [#124 Add support for other media types to popularity calculations](https://github.com/WordPress/openverse-catalog/pull/124)
- [#120 Clean Wikimedia item titles](https://github.com/WordPress/openverse-catalog/pull/120)
- [#119 Ingest wikimedia images marked with CC0 and PDM](https://github.com/WordPress/openverse-catalog/pull/119)
- [#118 Set default output dir for commoncrawl](https://github.com/WordPress/openverse-catalog/pull/118)
- [#116 Update issue templates](https://github.com/WordPress/openverse-catalog/pull/116)
- [#113 [API integration] Add Jamendo provider API script](https://github.com/WordPress/openverse-catalog/pull/113)
- [#112 Add support for other media types to popularity calculations](https://github.com/WordPress/openverse-catalog/pull/112)
- [#111 Add Audio to the database](https://github.com/WordPress/openverse-catalog/pull/111)
- [#110 Extract media type from staged tsv file name for loader](https://github.com/WordPress/openverse-catalog/pull/110)
- [#108 Remove logging of url rewriting when not rewritten](https://github.com/WordPress/openverse-catalog/pull/108)
- [#104 Run release drafter action on push to main branch](https://github.com/WordPress/openverse-catalog/pull/104)
- [#103 Fix pep8 violations](https://github.com/WordPress/openverse-catalog/pull/103)
- [#100 Remove mutable parameters in provider api scripts](https://github.com/WordPress/openverse-catalog/pull/100)
- [#98 Run ci on main push only](https://github.com/WordPress/openverse-catalog/pull/98)
- [#93 Create a Provider API script template](https://github.com/WordPress/openverse-catalog/pull/93)
- [#91 Fix the dependency version conflict](https://github.com/WordPress/openverse-catalog/pull/91)
- [#90 Ensure Docker loads local_postgres sql scripts in correct order](https://github.com/WordPress/openverse-catalog/pull/90)
- [#85 Add AudioStorage entity](https://github.com/WordPress/openverse-catalog/pull/85)
- [#83 Extract MediaStorage entity as parent to ImageStore](https://github.com/WordPress/openverse-catalog/pull/83)
- [#81 Fix failing test, improve import readability](https://github.com/WordPress/openverse-catalog/pull/81)

rendered:

openverse

openverse-frontend

openverse-api

openverse-catalog

Copy link
Member

@dhruvkb dhruvkb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if it's too early to review but this is exciting stuff!

  • It'd be nice to have the package.json inside the js/ folder as there's also a python/ directory at the same level. Keeps code that cannot interact with each other separated.
  • It'll also be cool to spin this off into a separate action and run it from our workflows. I think a lot more people can find it useful (and/or contribute to it) that way.

Some more feedback:

js/migrate_issues.js Outdated Show resolved Hide resolved
@dhruvkb
Copy link
Member

dhruvkb commented Aug 14, 2021

On a lighter note, this is beautiful.

Screenshot 2021-08-14 at 7 17 28 AM

Getting some xkcd vibes.

@zackkrida
Copy link
Member Author

haha way to early to review this. I'm going to squash the commits. Also, it doesn't work—the api doesn't even allow the querying I want regarding close dates.

@zackkrida
Copy link
Member Author

Okay now it works @dhruvkb, filtering the issues by closed_at. And even more functional hahahaha. look at this!

mapAwait(fetchRepoIssues)(repos)
  .then(mapAwait(getJSON))
  .then(map(filter(filterClosed)))
  .then(zipObj(repos))
  .then(compose(map(branch(logRepo, map(logIssue))), entries))
  .catch(console.error)

@zackkrida zackkrida marked this pull request as ready for review August 14, 2021 02:52
@zackkrida zackkrida requested a review from a team as a code owner August 14, 2021 02:52
@zackkrida zackkrida requested review from krysal and dhruvkb August 14, 2021 02:52
@dhruvkb
Copy link
Member

dhruvkb commented Aug 14, 2021

I had another idea! Can we get an API from WordPress and post this changelog to either the Team Openverse P2 or the Make site? That would give us something very close to a snaps post, but autogenerated.

js/src/weekly_report.js Show resolved Hide resolved
fetch(url, {
headers: {
Authorization:
'Basic emFja2tyaWRhOmdocF9JVHdJM2ZGUkVqSjZkOWRBUFFBOE5pWktwNlU0dXkxRTVFak8=',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A key for is not needed for reading public resources.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is useful locally however, for the purpose of rate limiting. Maybe I'll make it an optional script input.

@zackkrida
Copy link
Member Author

I had another idea! Can we get an API from WordPress and post this changelog to either the Team Openverse P2 or the Make site? That would give us something very close to a snaps post, but autogenerated.

Yeah, we could definitely look at doing this for the "A Week in Openverse" make posts. The idea is that Snaps will be fewer, deeper looks at things, with a focus on the visual component, and the make post is more of a formal changelog. Sometimes I'd be nice to manually include some of the Snap information in the Make posts, so we could just edit the posts after our automation publishes them.

Copy link
Contributor

@obulat obulat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, of course, if you've done cleaning up :)

@zackkrida zackkrida changed the title Sloppy weekly report script Weekly report script Aug 20, 2021
@zackkrida zackkrida requested a review from dhruvkb August 26, 2021 17:30
@zackkrida
Copy link
Member Author

@dhruvkb could you re-review this soon and check out the new comment replies?
Also, I created a new issue for the automation of the script: #55

@dhruvkb
Copy link
Member

dhruvkb commented Aug 26, 2021

Sure @zackkrida, I'll review it first thing tomorrow. Sorry to keep you waiting this long.

@zackkrida
Copy link
Member Author

Sure @zackkrida, I'll review it first thing tomorrow. Sorry to keep you waiting this long.

No worries at all, I left you hanging on some answers!

Copy link
Member

@dhruvkb dhruvkb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me 👍. I love the functional utils! I think I'm going to keep a copy for some personal use.

We'll definitely refactor when the number of JS scripts increases and necessitates that.

@zackkrida zackkrida merged commit 6a1c820 into main Aug 27, 2021
@zackkrida zackkrida deleted the weekly-report branch August 27, 2021 13:58
dhruvkb pushed a commit that referenced this pull request Feb 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants