This repository has been archived by the owner on Aug 4, 2023. It is now read-only.
Add Audio to the database #111
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tore Signed-off-by: Olga Bulat <[email protected]>
This reverts commit f3785d0 Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
# Conflicts: # src/cc_catalog_airflow/dags/common/storage/image.py # src/cc_catalog_airflow/dags/common/storage/test_image.py
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
obulat
requested review from
zackkrida and
dhruvkb
and removed request for
a team
Signed-off-by: Olga Bulat <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
zackkrida
previously approved these changes
Jun 25, 2021
👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 |
Signed-off-by: Olga Bulat <[email protected]>
36 tasks
Signed-off-by: Olga Bulat <[email protected]>
dhruvkb
previously approved these changes
Jun 29, 2021
krysal
reviewed
Jun 29, 2021
This was referenced Apr 17, 2023
Closed
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to WordPress/openverse#19
This is a cleaned-up version of #64 part related to saving the Audio data to the database. And #69 :)
This PR:
creates a new Audio database. The
sqlscripts for creation of the Postgres tables were added to bothlocal_postgresfolder that is used for local testing, and theopenledger_sqlfolder which is [probably] used in production.Reorders the AudioStore columns so that audio-specific columns are all at the end. This makes it easier to write and read the data, and to test. It would be nice to do the same for ImageStore, however we have many TSV files that use the columns in the current order, so we would need to detect the TSV file structure and process it accordingly.
adds media type parameter to the loader functions so that they can select correct list of columns to write to the database.
adds audio-specific SQL queries to
loader/sql.pyfor creating audio tables, upserting and overwriting audio in the database.Added on June 30:
watermarkedcolumn from image-specific columns to a list of columns common for all media types, as per API models, as suggested by @krysal .ingestion_typecolumn value for all media based onsourcevalue.There are three different places where data is added or overwritten in the database, and in each place the columns are listed separately because different columns are affected. I would like to have this process abstracted in the future, so that we don't have to update column lists in all three places.
Signed-off-by: Olga Bulat [email protected]