Add a documentation page describing the different parts of the Openverse stack #3786

zealshah29 · 2024-02-12T06:50:47Z

Fixes

Description

I have added a documentation page on the docs site that goes further in-depth about the various pieces of the stack, with references to the related documentation and code for each entity.

For each of the following sections of the stack listed, we have:

Link to the folder for the service on GitHub
Link to the documentation for the service on docs.openverse.org
Primary language(s) for the project
List of all of the tools/technologies/frameworks the services use along with links to each of its documentation

For example for the API, we have (respectively):

summary of the service
Code
Documentation
Language: Python
Tools & Frameworks:
- Django REST framework

Testing Instructions

Checklist

My pull request has a descriptive title (not a vague title likeUpdate index.md).
My pull request targets the default branch of the repository (main) or a parent feature branch.
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no visible errors.
I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

dhruvkb

@zealshah29 this is a great start, thanks for the contributions. I do have some notes that I've elaborated on below.

You need to lint the file so that the CI check for linting can pass. You can do so using the just lint recipe. Linting will also run automatically from then on as the lint recipe will download pre-commit and set up the Git hooks for you.
The newly added stack.md file needs to be added to the table of contents, which present is in the file https://github.com/WordPress/openverse/blob/3e57bfaff03d44b97e73ef5794ef0eda40b33749/documentation/general/index.md.
Automation is not technically a part of the stack as it's more concerned with general repository management. I would drop it but I'll defer to other maintainers for their views.
The one-line descriptions for the stack layers can be found in the main README.md file, it'll be good to re-use those here.
There are other components in our stack that are more general
purpose and shared between these layers. These include
- cache (Redis)
- upstream and API database (PostgreSQL)
- anaytics (third-party, Plausible)
IMO these can, and should, be included in their own separate subsection.

dhruvkb · 2024-02-12T16:06:49Z

documentation/general/stack.md

+   - [Documentation](https://docs.openverse.org/api/index.html)
+   - Language: Python
+   - Tools and Framework:
+     - [Django REST Framework](https://www.django-rest-framework.org/)


Also include Django above this.

Shall I add it to line number 25?

Yes, that would be great, thanks!

@dhruvkb do you have an opinion about the docs links being pinned to a version? (5.0 in this case?) Not sure there's anything we can do about that 🤔

Do I need to make any changes related to the link over here?

dhruvkb · 2024-02-12T16:07:53Z

documentation/general/stack.md

+   - Language: Python
+   - Tools and Framework:
+     - [Elasticsearch](https://www.elastic.co/guide/index.html)
+     - [Docker](https://docs.docker.com/)


Docker is basically a dev + deployment mechanism and not a part of the stack's technologies so I recommend dropping that.

dhruvkb · 2024-02-12T16:09:46Z

documentation/general/stack.md

+   - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt.
+   - [Code](https://github.com/WordPress/openverse/blob/main/frontend)
+   - [Documentation](https://docs.openverse.org/frontend/index.html)
+   - Language: JavaScript, HTML, CSS


HTML and CSS are kind-of universal to frontend dev so we can skip them. Also TypeScript is more appropriate here.

Suggested change

- Language: JavaScript, HTML, CSS

- Language: TypeScript / Node.js

dhruvkb · 2024-02-12T16:11:35Z

documentation/general/stack.md

+     - [Node.js](https://nodejs.org/docs/latest/api/)
+     - [Tailwind CSS](https://v2.tailwindcss.com/docs)


Node.js is neither a tool nor a framework. It's a JS runtime, so it feels incorrect to list here.

As for TailwindCSS, it is a very small, not-noteworthy slice of the number of technologies we use in the frontend.

dhruvkb · 2024-02-12T16:12:03Z

documentation/general/stack.md

+
+
+
+
+
+


These blank lines should be removed as the lint job in CI will complain.

zealshah29 · 2024-02-13T14:38:40Z

Hi, can you share the documentation for step-by-step guidance on how to lint the file?

You need to lint the file so that the CI check for linting can pass. You can do so using the just lint recipe. Linting will also run automatically from then on as the lint recipe will download pre-commit and set up the Git hooks for you.

Shall I create a separate subsection for this? Or shall I add them under tools and framework for each stack? Also do we have a folder on the GitHub for the following files along with its documentation?

There are other components in our stack that are more general
purpose and shared between these layers. These include

cache (Redis)

upstream and API database (PostgreSQL)

anaytics (third-party, Plausible)

IMO these can, and should, be included in their own separate subsection.

dhruvkb · 2024-02-13T17:26:11Z

Hi, can you share the documentation for step-by-step guidance on how to lint the file?

I search our docs and it turns out unfortunately, we do not have docs for that, which is a big miss on our end (and a potential opportunity for you to submit a PR for, once you've set it up for yourself).

Linting locally requires a bunch of tooling set up like just, Docker and Node.js. Would you be open to installing them? If so, please do so and then run just lint from the project root directory.

If you would like to not set them up, let us know and we'll lint the file for you.

Shall I create a separate subsection for this?

Yes, a separate section with a name like "External services" would be good to group these services.

Also do we have a folder on the GitHub for the following files along with its documentation?

All of our external services are in the https://github.com/WordPress/openverse/tree/b4b0cc939dec200efc3bf6db55b402c67ed678c7/docker directory.

Additionally, I missed some stacks like Documentation and some external services like Elasticsearch. Would be nice to include the documentation for them as well.

Thanks again for this contribution and for being conducive to my feedback.

zealshah29 · 2024-02-14T11:44:01Z

I search our docs and it turns out unfortunately, we do not have docs for that, which is a big miss on our end (and a potential opportunity for you to submit a PR for, once you've set it up for yourself).

Linting locally requires a bunch of tooling set up like just, Docker and Node.js. Would you be open to installing them? If so, please do so and then run just lint from the project root directory.

If you would like to not set them up, let us know and we'll lint the file for you.

Okay, I shall do the setup and try that on my own. In case I need help, I'll let you know.

Yes, a separate section with a name like "External services" would be good to group these services.

Okay

All of our external services are in the https://github.com/WordPress/openverse/tree/b4b0cc939dec200efc3bf6db55b402c67ed678c7/docker directory.

Additionally, I missed some stacks like Documentation and some external services like Elasticsearch. Would be nice to include the documentation for them as well.

Okay sure will go through it. So I'm planning to add them in the following manner:

External Services:

cache: Redis
upstream and API database: PostgreSQL
analytics: third-party, Plausible
...
The tools and frameworks used will be linked to their respective documentation. Would that be good?

Thanks again for this contribution and for being conducive to my feedback.

Thank you for your guidance and support.

zealshah29 · 2024-02-14T11:55:32Z

Automation is not technically a part of the stack as it's more concerned with general repository management. I would drop it but I'll defer to other maintainers for their views.

Also, shall I remove Automations from stack.md?

zealshah29 · 2024-02-18T06:59:45Z

I search our docs and it turns out unfortunately, we do not have docs for that, which is a big miss on our end (and a potential opportunity for you to submit a PR for, once you've set it up for yourself).

Linting locally requires a bunch of tooling set up like just, Docker and Node.js. Would you be open to installing them? If so, please do so and then run just lint from the project root directory.

If you would like to not set them up, let us know and we'll lint the file for you.

Hi so I tried installing just, but I am facing the following issue:

Also, I checked the justfile and I read that it's not compatible with Windows OS. Since I am using windows OS (git bash), is there any way to do linting on my system?

dhruvkb · 2024-02-18T07:16:36Z

@zealshah29 Openverse development is only supported for Linux or macOS, largely because none of the core maintainers have Windows. For Windows, we recommend WSL to get a Linux-like environment which is much more conducive for development.

However, in this case, we're happy to take over the PR and lint it ourselves.

zealshah29 · 2024-02-21T06:21:05Z

Hi, so I have updated the files as per the review.

stacimc

Thank you @zealshah29! Just a couple notes about the Catalog and ingestion server, and then we can get the linting fixed :)

documentation/general/stack.md

stacimc · 2024-02-23T01:28:51Z

documentation/general/stack.md

+   - [Documentation](https://docs.openverse.org/api/index.html)
+   - Language: Python
+   - Tools and Framework:
+     - [Django REST Framework](https://www.django-rest-framework.org/)


@dhruvkb do you have an opinion about the docs links being pinned to a version? (5.0 in this case?) Not sure there's anything we can do about that 🤔

zealshah29 · 2024-02-26T18:21:00Z

Thank you @zealshah29! Just a couple notes about the Catalog and ingestion server, and then we can get the linting fixed :)

Yes sure, I shall make the necessary changes and share an update once it's done 👍🏻

Co-authored-by: Staci Mullins <[email protected]>

Added PostgreSQL's documentation link.

zealshah29 · 2024-02-26T18:36:47Z

Thank you @zealshah29! Just a couple notes about the Catalog and ingestion server, and then we can get the linting fixed :)

Yes sure, I shall make the necessary changes and share an update once it's done 👍🏻

Hi, I have made the requested changes, could you please review it?

stacimc · 2024-02-26T19:14:27Z

Thanks for making those updates, @zealshah29! I think this is just about ready, but I've just noticed your PR is from your main branch. Could you please move your changes to a new feature branch and make a new PR from it, per the Openverse development guidelines? Let us know if you have any questions. Once that's done I'd be happy to push the linting changes for you and approve!

stacimc

On second thought, since this PR has been open for awhile and has a good bit of context, I'll approve as is! Please use a feature branch in the future, however.

Thanks again for your contribution!

zealshah29 · 2024-02-27T02:39:37Z

On second thought, since this PR has been open for awhile and has a good bit of context, I'll approve as is! Please use a feature branch in the future, however.

Thanks again for your contribution!

Thank you! Sure I will take care of that from next time.

openverse-bot · 2024-02-28T00:00:10Z

Based on the low urgency of this PR, the following reviewers are being gently reminded to review this PR:

@dhruvkb
This reminder is being automatically generated due to the urgency configuration.

Excluding weekend¹ days, this PR was ready for review 11 day(s) ago. PRs labelled with low urgency are expected to be reviewed within 5 weekday(s)².

@zealshah29, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings.

Specifically, Saturday and Sunday. ↩
For the purpose of these reminders we treat Monday - Friday as weekdays. Please note that the operation that generates these reminders runs at midnight UTC on Monday - Friday. This means that depending on your timezone, you may be pinged outside of the expected range. ↩

dhruvkb

Thanks for linting @stacimc. I agree with your reasoning to merge it and open separate issues later.

zealshah29 added 7 commits February 6, 2024 21:20

Create stack.md

0af33c0

Merge branch 'WordPress:main' into main

f720d10

Update stack.md

6b3da78

Update stack.md

963c1ae

Update stack.md

2d34d12

Update stack.md

cc0c368

Merge branch 'WordPress:main' into main

550892c

zealshah29 requested a review from a team as a code owner February 12, 2024 06:50

zealshah29 requested review from dhruvkb and stacimc February 12, 2024 06:50

obulat removed the 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work label Feb 12, 2024

dhruvkb requested changes Feb 12, 2024

View reviewed changes

Update stack.md

242e4ef

dhruvkb added 🧱 stack: documentation Related to Sphinx documentation and removed 🏷 status: label work required Needs proper labelling before it can be worked on labels Feb 15, 2024

zealshah29 added 4 commits February 15, 2024 19:01

Update stack.md

397655a

Update index.md

d224fc0

Update stack.md

d6f8fc2

Merge branch 'WordPress:main' into main

ff6bdc7

Merge branch 'WordPress:main' into main

6349f8f

sarayourfriend requested a review from dhruvkb February 23, 2024 00:34

stacimc requested changes Feb 23, 2024

View reviewed changes

zealshah29 and others added 2 commits February 26, 2024 23:57

Applied suggestions from code review

49ba95d

Co-authored-by: Staci Mullins <[email protected]>

Update stack.md

bf7a519

Added PostgreSQL's documentation link.

Lint

63b7b94

stacimc approved these changes Feb 26, 2024

View reviewed changes

dhruvkb changed the title ~~Added a documentation page describing the different parts of the Openverese stack.~~ Add a documentation page describing the different parts of the Openverse stack Feb 28, 2024

dhruvkb approved these changes Feb 28, 2024

View reviewed changes

dhruvkb merged commit b873150 into WordPress:main Feb 28, 2024
41 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a documentation page describing the different parts of the Openverse stack #3786

Add a documentation page describing the different parts of the Openverse stack #3786

zealshah29 commented Feb 12, 2024 •

edited

Loading

dhruvkb left a comment

dhruvkb Feb 12, 2024

zealshah29 Feb 13, 2024

dhruvkb Feb 13, 2024

stacimc Feb 23, 2024

zealshah29 Feb 26, 2024

dhruvkb Feb 12, 2024

zealshah29 Feb 13, 2024

dhruvkb Feb 12, 2024

zealshah29 Feb 13, 2024

dhruvkb Feb 12, 2024

zealshah29 Feb 13, 2024

dhruvkb Feb 12, 2024

zealshah29 Feb 13, 2024

zealshah29 commented Feb 13, 2024

dhruvkb commented Feb 13, 2024 •

edited

Loading

zealshah29 commented Feb 14, 2024

zealshah29 commented Feb 14, 2024

zealshah29 commented Feb 18, 2024

dhruvkb commented Feb 18, 2024

zealshah29 commented Feb 21, 2024

stacimc left a comment

stacimc Feb 23, 2024

zealshah29 commented Feb 26, 2024

zealshah29 commented Feb 26, 2024

stacimc commented Feb 26, 2024

stacimc left a comment

zealshah29 commented Feb 27, 2024

openverse-bot commented Feb 28, 2024

dhruvkb left a comment

	- Language: JavaScript, HTML, CSS
	- Language: TypeScript / Node.js

		- [Node.js](https://nodejs.org/docs/latest/api/)
		- [Tailwind CSS](https://v2.tailwindcss.com/docs)

Add a documentation page describing the different parts of the Openverse stack #3786

Add a documentation page describing the different parts of the Openverse stack #3786

Conversation

zealshah29 commented Feb 12, 2024 • edited Loading

Fixes

Description

Testing Instructions

Checklist

Developer Certificate of Origin

dhruvkb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zealshah29 commented Feb 13, 2024

dhruvkb commented Feb 13, 2024 • edited Loading

zealshah29 commented Feb 14, 2024

zealshah29 commented Feb 14, 2024

zealshah29 commented Feb 18, 2024

dhruvkb commented Feb 18, 2024

zealshah29 commented Feb 21, 2024

stacimc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zealshah29 commented Feb 26, 2024

zealshah29 commented Feb 26, 2024

stacimc commented Feb 26, 2024

stacimc left a comment

Choose a reason for hiding this comment

zealshah29 commented Feb 27, 2024

openverse-bot commented Feb 28, 2024

Footnotes

dhruvkb left a comment

Choose a reason for hiding this comment

zealshah29 commented Feb 12, 2024 •

edited

Loading

dhruvkb commented Feb 13, 2024 •

edited

Loading