Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make popularities.json dynamic instead of checked into git #4286

Conversation

peterbe
Copy link
Contributor

@peterbe peterbe commented Jul 22, 2021

Part of #4217

The idea is that we drive the popularities from /popularities.json instead. Reading it is always done via the getPopularities() function and it now hardcodes the location of the file.

If you're doing local development in yari with yarn dev you get:

info: Reusing exising /Users/peterbe/dev/MOZILLA/MDN/yari/popularities.json (Thu Jul 22 2021 14:34:37 GMT-0400 (Eastern Daylight Time)) for popularities.
info: Reset /Users/peterbe/dev/MOZILLA/MDN/yari/popularities.json by running: yarn tool popularities --refresh

which makes consecutive runs of yarn dev fast.

Then, when it bundles up yari as an NPM package, you're going to get a tarball that, at its root, contains a popularities.json which got created (every time) from the .github/workflows/npm-publish.yml. So when you run yari from within the mdn/content repo, you will find a node_modules/@mdn/yari/popularities.json which the node_modules/@mdn/yari/content/popularities.js can always reach.

This way, every new npm package will have a fresh popularities.json from S3. So writers, who use the autocomplete search on localhost:5000 will get the latest and greatest (or at least as fresh as the Athena job runs). And their yarn start will still be fast and not need to download from S3.

The reason this PR is only Part of #4217 is because once shipped to NPM we need to go into mdn/content and run git rm files/popularities.json.

@peterbe peterbe requested a review from fiji-flo July 22, 2021 19:03
@peterbe
Copy link
Contributor Author

peterbe commented Jul 22, 2021

One thing to think about is that this will require every CI run of yari to have to depend on that network download. But S3 is reliable and if it ever turns into an annoying delay, we can hack around it. Or inject a sample file specifically for the tests.

@peterbe
Copy link
Contributor Author

peterbe commented Jul 28, 2021

@fiji-flo Any chance you can review this, this week?

The https://github.com/mdn/content/blob/main/files/popularities.json file is now 3 months old.

@escattone escattone merged commit 9335f80 into mdn:main Aug 3, 2021
@peterbe peterbe deleted the 4217-make-popularitiesjson-dynamic-instead-of-checked-into-git branch August 3, 2021 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants