From 0af33c03f1969772618dc9caf9c8be3b25d71479 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Tue, 6 Feb 2024 21:20:19 +0530 Subject: [PATCH 01/12] Create stack.md --- documentation/general/stack.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 documentation/general/stack.md diff --git a/documentation/general/stack.md b/documentation/general/stack.md new file mode 100644 index 00000000000..1ccb0898d28 --- /dev/null +++ b/documentation/general/stack.md @@ -0,0 +1,6 @@ +- summary of the service +- Code +- Documentation +- Language: Python +- Tools & Frameworks: +- Django REST framework From 6b3da78efad1c4525e200a920e2c4913fb2f60f3 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Wed, 7 Feb 2024 23:18:56 +0530 Subject: [PATCH 02/12] Update stack.md --- documentation/general/stack.md | 37 ++++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index 1ccb0898d28..9e85422b639 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -1,6 +1,31 @@ -- summary of the service -- Code -- Documentation -- Language: Python -- Tools & Frameworks: -- Django REST framework +Summary of service: +- Catalog: Powered by Apache Airflow, it stores Openverse metadata efficiently. +- Ingestion Server: Syncs data from Catalog to API, ensuring freshness. +- API: Built with Django REST, it enables database querying for the frontend. +- Frontend: Vue/Nuxt-based search engine at openverse.org. +- Automation: Scripts automating Openverse workflows. + + + +Code: +- Catalog +- Ingestion server +- API +- Frontend +- Automations + + +Documentation: +- https://docs.openverse.org/catalog/index.html +- https://docs.openverse.org/ingestion_server/index.html +- https://docs.openverse.org/api/index.html +- https://docs.openverse.org/frontend/index.html +- https://docs.openverse.org/automations/index.html + +Language: +- Catalog: python +- Ingestion Server: python +- API: python +- Frontend: JavaScript, HTML, CSS +- Automation: Python, Node.js + From 963c1aee8018509808092f67d410300bdd4fa60f Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Sat, 10 Feb 2024 19:32:54 +0530 Subject: [PATCH 03/12] Update stack.md --- documentation/general/stack.md | 73 +++++++++++++++++++++------------- 1 file changed, 46 insertions(+), 27 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index 9e85422b639..e4c7df687de 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -1,31 +1,50 @@ -Summary of service: -- Catalog: Powered by Apache Airflow, it stores Openverse metadata efficiently. -- Ingestion Server: Syncs data from Catalog to API, ensuring freshness. -- API: Built with Django REST, it enables database querying for the frontend. -- Frontend: Vue/Nuxt-based search engine at openverse.org. -- Automation: Scripts automating Openverse workflows. - - - -Code: -- Catalog -- Ingestion server -- API -- Frontend -- Automations +# Stack +This page is your gateway to understanding our diverse technology stack. Explore code repositories on GitHub, access detailed documentation, and learn about the tools, frameworks, and languages driving each stack of the Openverse platform. + +1. Catalog + - Powered by Apache Airflow, it stores Openverse metadata efficiently. + - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - [Documentation]() + - Language: Python + - Tools and Framework: + - [Apache Airflow]() +2. Ingestion Server + - Powered by Apache Airflow, it stores Openverse metadata efficiently. + - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - [Documentation]() + - Language: Python + - Tools and Framework: + - [Elasticsearch]() + - [Docker]() +3. API + - Powered by Apache Airflow, it stores Openverse metadata efficiently. + - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - [Documentation]() + - Language: Python + - Tools and Framework: + - [Django REST Framework]() + - [Docker]() +4. Frontend + - Powered by Apache Airflow, it stores Openverse metadata efficiently. + - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - [Documentation]() + - Language: JavaScript, HTML, CSS + - Tools and Framework: + - [Nuxt.js]() + - [Vue.js]() + - [Node.js]() + - [Tailwind CSS]() +5. Automation + - Powered by Apache Airflow, it stores Openverse metadata efficiently. + - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - [Documentation]() + - Language: Python + - Tools and Framework: + - [Node.js]() + + + -Documentation: -- https://docs.openverse.org/catalog/index.html -- https://docs.openverse.org/ingestion_server/index.html -- https://docs.openverse.org/api/index.html -- https://docs.openverse.org/frontend/index.html -- https://docs.openverse.org/automations/index.html -Language: -- Catalog: python -- Ingestion Server: python -- API: python -- Frontend: JavaScript, HTML, CSS -- Automation: Python, Node.js From 2d34d125430825ebe7f235c5ab40eafa2a93fba8 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Sat, 10 Feb 2024 19:45:00 +0530 Subject: [PATCH 04/12] Update stack.md --- documentation/general/stack.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index e4c7df687de..0a84f488b5f 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -5,29 +5,29 @@ This page is your gateway to understanding our diverse technology stack. Explore 1. Catalog - Powered by Apache Airflow, it stores Openverse metadata efficiently. - [Code](https://github.com/WordPress/openverse/blob/main/catalog) - - [Documentation]() + - [Documentation](https://docs.openverse.org/catalog/index.html) - Language: Python - Tools and Framework: - - [Apache Airflow]() + - [Apache Airflow](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datacatalog.html) 2. Ingestion Server - - Powered by Apache Airflow, it stores Openverse metadata efficiently. - - [Code](https://github.com/WordPress/openverse/blob/main/catalog) - - [Documentation]() + - The mechanism for refreshing the data from the catalog to the API. + - [Code](https://github.com/WordPress/openverse/blob/main/ingestion_server) + - [Documentation](https://docs.openverse.org/ingestion_server/index.html) - Language: Python - Tools and Framework: - - [Elasticsearch]() - - [Docker]() + - [Elasticsearch](https://www.elastic.co/guide/index.html) + - [Docker](https://docs.docker.com/) 3. API - - Powered by Apache Airflow, it stores Openverse metadata efficiently. - - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - Built with Django REST, it enables database querying for the Frontend. + - [Code](https://github.com/WordPress/openverse/blob/main/api) - [Documentation]() - Language: Python - Tools and Framework: - [Django REST Framework]() - [Docker]() 4. Frontend - - Powered by Apache Airflow, it stores Openverse metadata efficiently. - - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt. + - [Code]() - [Documentation]() - Language: JavaScript, HTML, CSS - Tools and Framework: @@ -36,8 +36,8 @@ This page is your gateway to understanding our diverse technology stack. Explore - [Node.js]() - [Tailwind CSS]() 5. Automation - - Powered by Apache Airflow, it stores Openverse metadata efficiently. - - [Code](https://github.com/WordPress/openverse/blob/main/catalog) + - Scripts used for various workflows around Openverse repositories and processes. + - [Code]() - [Documentation]() - Language: Python - Tools and Framework: From cc0c36834d28197c7a6881a974797746c8456fc2 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Mon, 12 Feb 2024 11:52:35 +0530 Subject: [PATCH 05/12] Update stack.md --- documentation/general/stack.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index 0a84f488b5f..854d0da5ffc 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -20,28 +20,28 @@ This page is your gateway to understanding our diverse technology stack. Explore 3. API - Built with Django REST, it enables database querying for the Frontend. - [Code](https://github.com/WordPress/openverse/blob/main/api) - - [Documentation]() + - [Documentation](https://docs.openverse.org/api/index.html) - Language: Python - Tools and Framework: - - [Django REST Framework]() - - [Docker]() + - [Django REST Framework](https://www.django-rest-framework.org/) + - [Docker](https://docs.docker.com/) 4. Frontend - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt. - - [Code]() - - [Documentation]() + - [Code](https://github.com/WordPress/openverse/blob/main/frontend) + - [Documentation](https://docs.openverse.org/frontend/index.html) - Language: JavaScript, HTML, CSS - Tools and Framework: - - [Nuxt.js]() - - [Vue.js]() - - [Node.js]() - - [Tailwind CSS]() + - [Nuxt.js](https://nuxt.com/docs/getting-started/introduction) + - [Vue.js](https://vuejs.org/guide/introduction.html) + - [Node.js](https://nodejs.org/docs/latest/api/) + - [Tailwind CSS](https://v2.tailwindcss.com/docs) 5. Automation - Scripts used for various workflows around Openverse repositories and processes. - - [Code]() - - [Documentation]() + - [Code](https://github.com/WordPress/openverse/tree/main/automations) + - [Documentation](https://docs.openverse.org/automations/index.html) - Language: Python - Tools and Framework: - - [Node.js]() + - [Node.js](https://nodejs.org/docs/latest/api/) From 242e4ef83471c8d7a63e0fbb1196f6b6b3cc3a10 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Wed, 14 Feb 2024 17:25:53 +0530 Subject: [PATCH 06/12] Update stack.md --- documentation/general/stack.md | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index 854d0da5ffc..281fb9a272a 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -16,7 +16,6 @@ This page is your gateway to understanding our diverse technology stack. Explore - Language: Python - Tools and Framework: - [Elasticsearch](https://www.elastic.co/guide/index.html) - - [Docker](https://docs.docker.com/) 3. API - Built with Django REST, it enables database querying for the Frontend. - [Code](https://github.com/WordPress/openverse/blob/main/api) @@ -24,27 +23,16 @@ This page is your gateway to understanding our diverse technology stack. Explore - Language: Python - Tools and Framework: - [Django REST Framework](https://www.django-rest-framework.org/) - - [Docker](https://docs.docker.com/) 4. Frontend - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt. - [Code](https://github.com/WordPress/openverse/blob/main/frontend) - [Documentation](https://docs.openverse.org/frontend/index.html) - - Language: JavaScript, HTML, CSS + - Language: TypeScript / Node.js - Tools and Framework: - [Nuxt.js](https://nuxt.com/docs/getting-started/introduction) - [Vue.js](https://vuejs.org/guide/introduction.html) - - [Node.js](https://nodejs.org/docs/latest/api/) - - [Tailwind CSS](https://v2.tailwindcss.com/docs) 5. Automation - Scripts used for various workflows around Openverse repositories and processes. - [Code](https://github.com/WordPress/openverse/tree/main/automations) - [Documentation](https://docs.openverse.org/automations/index.html) - Language: Python - - Tools and Framework: - - [Node.js](https://nodejs.org/docs/latest/api/) - - - - - - From 397655a32b8ebaeaeacf0c63b13fe9f4a9b400b2 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Thu, 15 Feb 2024 19:01:34 +0530 Subject: [PATCH 07/12] Update stack.md --- documentation/general/stack.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index 281fb9a272a..f1c11a25327 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -31,8 +31,4 @@ This page is your gateway to understanding our diverse technology stack. Explore - Tools and Framework: - [Nuxt.js](https://nuxt.com/docs/getting-started/introduction) - [Vue.js](https://vuejs.org/guide/introduction.html) -5. Automation - - Scripts used for various workflows around Openverse repositories and processes. - - [Code](https://github.com/WordPress/openverse/tree/main/automations) - - [Documentation](https://docs.openverse.org/automations/index.html) - - Language: Python +5. From d224fc037221e892942d6d0b4eb093e1a11819cc Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Thu, 15 Feb 2024 19:02:30 +0530 Subject: [PATCH 08/12] Update index.md --- documentation/general/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/general/index.md b/documentation/general/index.md index 76fb15ce097..ffef65b82d4 100644 --- a/documentation/general/index.md +++ b/documentation/general/index.md @@ -12,5 +12,6 @@ zero_downtime_database_management deployment logging https +stack ``` From d6f8fc2e467f7640b4860506b4e618b559127d08 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Thu, 15 Feb 2024 19:28:16 +0530 Subject: [PATCH 09/12] Update stack.md --- documentation/general/stack.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index f1c11a25327..addd429af05 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -3,32 +3,37 @@ This page is your gateway to understanding our diverse technology stack. Explore code repositories on GitHub, access detailed documentation, and learn about the tools, frameworks, and languages driving each stack of the Openverse platform. 1. Catalog - - Powered by Apache Airflow, it stores Openverse metadata efficiently. + - The Apache Airflow-powered system for downloading and storing Openverse's metadata - [Code](https://github.com/WordPress/openverse/blob/main/catalog) - [Documentation](https://docs.openverse.org/catalog/index.html) - Language: Python - Tools and Framework: - [Apache Airflow](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datacatalog.html) 2. Ingestion Server - - The mechanism for refreshing the data from the catalog to the API. + - The mechanism for refreshing the data from the catalog to the API - [Code](https://github.com/WordPress/openverse/blob/main/ingestion_server) - [Documentation](https://docs.openverse.org/ingestion_server/index.html) - Language: Python - Tools and Framework: - [Elasticsearch](https://www.elastic.co/guide/index.html) 3. API - - Built with Django REST, it enables database querying for the Frontend. + - The Django REST API for querying the database, used by the frontend - [Code](https://github.com/WordPress/openverse/blob/main/api) - [Documentation](https://docs.openverse.org/api/index.html) - Language: Python - Tools and Framework: + - [Django](https://docs.djangoproject.com/en/5.0/) - [Django REST Framework](https://www.django-rest-framework.org/) 4. Frontend - - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt. + - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt - [Code](https://github.com/WordPress/openverse/blob/main/frontend) - [Documentation](https://docs.openverse.org/frontend/index.html) - Language: TypeScript / Node.js - Tools and Framework: - [Nuxt.js](https://nuxt.com/docs/getting-started/introduction) - [Vue.js](https://vuejs.org/guide/introduction.html) -5. +5. External Services + - Cache: [Redis](https://redis.io/docs/about/) + - Upstream and API database: [PostgreSQL](https://www.postgresql.org/docs/) + - Analytics: third-party, [Plausible](https://plausible.io/docs) + - Search: [Elasticsearch](https://www.elastic.co/guide/index.html) From 49ba95dad3c446845eaa698fc8d7ba653cbcc062 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Mon, 26 Feb 2024 23:57:01 +0530 Subject: [PATCH 10/12] Applied suggestions from code review Co-authored-by: Staci Mullins <63313398+stacimc@users.noreply.github.com> --- documentation/general/stack.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index addd429af05..ccd503ba9d1 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -8,9 +8,9 @@ This page is your gateway to understanding our diverse technology stack. Explore - [Documentation](https://docs.openverse.org/catalog/index.html) - Language: Python - Tools and Framework: - - [Apache Airflow](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datacatalog.html) + - [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html) 2. Ingestion Server - - The mechanism for refreshing the data from the catalog to the API + - The mechanism for refreshing the data from the catalog to the API DB and Elasticsearch indices - [Code](https://github.com/WordPress/openverse/blob/main/ingestion_server) - [Documentation](https://docs.openverse.org/ingestion_server/index.html) - Language: Python From bf7a5196f06fb9801d2b56f952f4c1d0cff0ca54 Mon Sep 17 00:00:00 2001 From: Zeal Shah <75566085+zealshah29@users.noreply.github.com> Date: Tue, 27 Feb 2024 00:02:20 +0530 Subject: [PATCH 11/12] Update stack.md Added PostgreSQL's documentation link. --- documentation/general/stack.md | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index ccd503ba9d1..820a8942f31 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -16,6 +16,7 @@ This page is your gateway to understanding our diverse technology stack. Explore - Language: Python - Tools and Framework: - [Elasticsearch](https://www.elastic.co/guide/index.html) + - [PostgreSQL](https://www.postgresql.org/docs/) 3. API - The Django REST API for querying the database, used by the frontend - [Code](https://github.com/WordPress/openverse/blob/main/api) From 63b7b94bfdec57e2968d83e88fc4dff8604b2d3b Mon Sep 17 00:00:00 2001 From: Staci Cooper Date: Mon, 26 Feb 2024 12:18:45 -0800 Subject: [PATCH 12/12] Lint --- documentation/general/stack.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/documentation/general/stack.md b/documentation/general/stack.md index 820a8942f31..ea1d215dbc8 100644 --- a/documentation/general/stack.md +++ b/documentation/general/stack.md @@ -1,16 +1,20 @@ # Stack -This page is your gateway to understanding our diverse technology stack. Explore code repositories on GitHub, access detailed documentation, and learn about the tools, frameworks, and languages driving each stack of the Openverse platform. +This page is your gateway to understanding our diverse technology stack. Explore +code repositories on GitHub, access detailed documentation, and learn about the +tools, frameworks, and languages driving each stack of the Openverse platform. 1. Catalog - - The Apache Airflow-powered system for downloading and storing Openverse's metadata + - The Apache Airflow-powered system for downloading and storing Openverse's + metadata - [Code](https://github.com/WordPress/openverse/blob/main/catalog) - [Documentation](https://docs.openverse.org/catalog/index.html) - Language: Python - Tools and Framework: - [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html) 2. Ingestion Server - - The mechanism for refreshing the data from the catalog to the API DB and Elasticsearch indices + - The mechanism for refreshing the data from the catalog to the API DB and + Elasticsearch indices - [Code](https://github.com/WordPress/openverse/blob/main/ingestion_server) - [Documentation](https://docs.openverse.org/ingestion_server/index.html) - Language: Python @@ -26,7 +30,8 @@ This page is your gateway to understanding our diverse technology stack. Explore - [Django](https://docs.djangoproject.com/en/5.0/) - [Django REST Framework](https://www.django-rest-framework.org/) 4. Frontend - - The public search engine at [openverse.org](https://openverse.org/), built with Vue and Nuxt + - The public search engine at [openverse.org](https://openverse.org/), built + with Vue and Nuxt - [Code](https://github.com/WordPress/openverse/blob/main/frontend) - [Documentation](https://docs.openverse.org/frontend/index.html) - Language: TypeScript / Node.js