Check language metadata #385

catreedle · 2024-10-16T13:57:42Z

Contributor checklist

This pull request is on a separate branch and not the main branch

Description

This PR introduces check_language_metadata.py with the functionality:

Check and log the differences between the languages in languages_metadata.json and language_data_extraction.
Validate that each language and sub-language in language_metadata.json has the property qid and iso.

This code has been tested locally.

Related issue

Add workflow to check project metadata #340

…e/Scribe-Data into check_language_metadata

github-actions · 2024-10-16T13:58:11Z

Thank you for the pull request!

The Scribe team will do our best to address your contribution as soon as we can. The following is a checklist for maintainers to make sure this process goes as well as possible. Feel free to address the points below yourself in further commits if you realize that actions are needed :)

If you're not already a member of our public Matrix community, please consider joining! We'd suggest using Element as your Matrix client, and definitely join the General and Data rooms once you're in. Also consider joining our bi-weekly Saturday dev syncs. It'd be great to have you!

Maintainer checklist

The linting and formatting workflow within the PR checks do not indicate new errors in the files changed
The CHANGELOG has been updated with a description of the changes for the upcoming release and the corresponding issue (if necessary)

DeleMike · 2024-10-16T14:11:14Z

src/scribe_data/check/check_language_metadata.py

+LANGUAGE_DATA_EXTRACTION_DIR = Path(__file__).parent.parent / "language_data_extraction"
+
+LANGUAGE_METADATA_FILE = (
+    Path(__file__).parent.parent / "resources" / "language_metadata.json"
+)
+
+DATA_TYPE_METADATA_FILE = (
+    Path(__file__).parent.parent / "resources" / "data_type_metadata.json"
+)


@catreedle you can get these from src/scribe_data/cli/cli_utils.py - they already exist there

DeleMike · 2024-10-16T14:12:49Z

src/scribe_data/check/check_language_metadata.py

+try:
+    with LANGUAGE_METADATA_FILE.open("r", encoding="utf-8") as file:
+        language_metadata = json.load(file)
+        languages_in_metadata = {
+            lang["language"]: {"iso": lang["iso"], "qid": lang["qid"]}
+            for lang in language_metadata["languages"]
+        }  # current language metadata
+
+        # languages_in_metadata = { # proposed language metadata
+        #     key.lower(): value for key, value in language_metadata.items()
+        # }  # Normalize keys to lowercase for case-insensitive comparison
+
+except (IOError, json.JSONDecodeError) as e:
+    print(f"Error reading language metadata: {e}")
+
+try:
+    with DATA_TYPE_METADATA_FILE.open("r", encoding="utf-8") as file:
+        data_type_metadata = json.load(file)
+        all_data_types = tuple(data_type_metadata.keys())
+
+except (IOError, json.JSONDecodeError) as e:


Also, the cli_utils.py has loaded the language_metadata.json and data_type_metadata.json files for us. You could ignore loading it in here

see the cli_utils.py contents

does it mean we can directly use it? how?

Yes, just call it directly. You can experiment and see for yourself 😄

Thankss.. I found it 😊

DeleMike · 2024-10-16T14:16:03Z

src/scribe_data/check/check_language_metadata.py

+if __name__ == "__main__":
+    check_language_metadata()


If this is for a test, then it's fine. but we will be calling this in check_project_metadata.yaml file so no need for this :)

thank you! good to know 😊

andrewtavis · 2024-10-16T22:45:55Z

One minor thing here, @catreedle: Could we get consistent function docstirng, and ones similar to what we have in this PR in get_missing_languages? :)

andrewtavis · 2024-10-16T22:46:25Z

Making the docstring like that means we can use autodoc in the documentation 😊

DeleMike · 2024-10-16T22:47:21Z

hey @catreedle , I'm sorry if I have confused you a bit :(

if __name__ == "__main__":
    check_language_metadata()

I just checked and the code you added earlier is quite important if we are gonna be using this file for our workflow. Can you please add it back?

Ty!

catreedle · 2024-10-17T00:10:10Z

One minor thing here, @catreedle: Could we get consistent function docstirng, and ones similar to what we have in this PR in get_missing_languages? :)

on it :)

catreedle · 2024-10-17T00:10:46Z

hey @catreedle , I'm sorry if I have confused you a bit :(
if __name__ == "__main__":
    check_language_metadata()
I just checked and the code you added earlier is quite important if we are gonna be using this file for our workflow. Can you please add it back?

Ty!

sure. no worries :)

catreedle · 2024-10-17T00:30:41Z

updated the docstring. let me know if there's amiss @andrewtavis :)

src/scribe_data/check/check_language_metadata.py

…e/Scribe-Data into check_language_metadata

andrewtavis · 2024-10-18T01:14:14Z

Quick note being sent to all the testing PRs, if updates are needed now that #402 has been merged, then it'd be great to get those updates to the branch :) If no updates are needed, then let me know 😊

catreedle · 2024-10-18T12:58:21Z

Quick note being sent to all the testing PRs, if updates are needed now that #402 has been merged, then it'd be great to get those updates to the branch :) If no updates are needed, then let me know 😊

Hi! I've made the necessary updates after the #402 merge. Everything should be good to go now 😊.

OmarAI2003 · 2024-10-18T13:26:07Z

src/scribe_data/check/check_language_metadata.py

+            If any missing languages or properties are found, the function exits the script with a status code of 1.
+    """
+    languages_in_metadata = {key.lower(): value for key, value in _languages.items()}
+


Does this line
languages_in_metadata = {key.lower(): value for key, value in _languages.items()}
give all the languages in _languages, including the sublanguages?

yes. but it doesn't lowercase all the sublanguages if any is mistakenly written, I just noticed.

This is graet. There is also a helper function called list_all_languages at the bottom of utils.py that lists all the languages in the JSON file. You can make use of it here to check for sublanguages; it would be helpful.

Thanks for the suggestion! Could you explain further how best to use it in this context? I want to include parent languages so I convert the languages directory to match the format of language_metadata.json for easier comparison. Since list_all_languages doesn’t show the parent languages, I’m unsure how to use it effectively for my comparison.

andrewtavis

I'm bringing this down and integrating the checks for isos and qids into the other workflows 😊 Thanks, @catreedle! :) Really great to have the support on the checks 🚀

catreedle · 2024-10-24T07:13:22Z

Thank you! @andrewtavis

catreedle and others added 9 commits October 14, 2024 22:57

check language metadata scribe-org#340

dd179c7

Merge branch 'scribe-org:main' into check_language_metadata

eb79e73

add sub_languages scribe-org#340

056d796

Merge branch 'scribe-org:main' into check_language_metadata

7149172

Merge branch 'check_language_metadata' of https://github.com/catreedl…

562f96a

…e/Scribe-Data into check_language_metadata

Fix: Add sub_languages only if they are not data types scribe-org#340

885dc94

validate language properties scribe-org#340

399dd37

Merge branch 'scribe-org:main' into check_language_metadata

e47bf55

Merge branch 'check_language_metadata' of https://github.com/catreedl…

e3f8d5f

…e/Scribe-Data into check_language_metadata

catreedle mentioned this pull request Oct 16, 2024

Add workflow to check project metadata #340

Closed

2 tasks

DeleMike reviewed Oct 16, 2024

View reviewed changes

import metadata from cli_utils

6808796

andrewtavis added the hacktoberfest-accepted Accepted as a part of Hacktoberfest label Oct 16, 2024

Merge branch 'scribe-org:main' into check_language_metadata

208fba7

Fix docstring for check_language_metadata.py

3ec3e74

DeleMike mentioned this pull request Oct 17, 2024

Moving from Old Language Metadata Structure to Support Sub-languages and Simplified JSON #402

Merged

1 task

OmarAI2003 suggested changes Oct 17, 2024

View reviewed changes

src/scribe_data/check/check_language_metadata.py Outdated Show resolved Hide resolved

catreedle and others added 3 commits October 18, 2024 00:00

update languages_in_metadata

3969f30

Merge branch 'scribe-org:main' into check_language_metadata

392977e

Merge branch 'check_language_metadata' of https://github.com/catreedl…

d238018

…e/Scribe-Data into check_language_metadata

catreedle and others added 2 commits October 18, 2024 14:01

Merge branch 'scribe-org:main' into check_language_metadata

122969d

import _languages from utils

0c87b59

OmarAI2003 reviewed Oct 18, 2024

View reviewed changes

OmarAI2003 mentioned this pull request Oct 20, 2024

Update list tests to not need to have the full output in the test #437

Closed

2 tasks

andrewtavis approved these changes Oct 24, 2024

View reviewed changes

andrewtavis merged commit 0fc2200 into scribe-org:main Oct 24, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check language metadata #385

Check language metadata #385

catreedle commented Oct 16, 2024

github-actions bot commented Oct 16, 2024 •

edited by andrewtavis

Loading

DeleMike Oct 16, 2024

DeleMike Oct 16, 2024

DeleMike Oct 16, 2024

catreedle Oct 16, 2024

DeleMike Oct 16, 2024

catreedle Oct 16, 2024

DeleMike Oct 16, 2024

catreedle Oct 16, 2024

andrewtavis commented Oct 16, 2024

andrewtavis commented Oct 16, 2024

DeleMike commented Oct 16, 2024

catreedle commented Oct 17, 2024

catreedle commented Oct 17, 2024

catreedle commented Oct 17, 2024

andrewtavis commented Oct 18, 2024

catreedle commented Oct 18, 2024

OmarAI2003 Oct 18, 2024

catreedle Oct 18, 2024

OmarAI2003 Oct 18, 2024 •

edited

Loading

catreedle Oct 18, 2024

andrewtavis left a comment

catreedle commented Oct 24, 2024

Check language metadata #385

Check language metadata #385

Conversation

catreedle commented Oct 16, 2024

Contributor checklist

Description

Related issue

github-actions bot commented Oct 16, 2024 • edited by andrewtavis Loading

Thank you for the pull request!

Maintainer checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewtavis commented Oct 16, 2024

andrewtavis commented Oct 16, 2024

DeleMike commented Oct 16, 2024

catreedle commented Oct 17, 2024

catreedle commented Oct 17, 2024

catreedle commented Oct 17, 2024

andrewtavis commented Oct 18, 2024

catreedle commented Oct 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OmarAI2003 Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewtavis left a comment

Choose a reason for hiding this comment

catreedle commented Oct 24, 2024

github-actions bot commented Oct 16, 2024 •

edited by andrewtavis

Loading

OmarAI2003 Oct 18, 2024 •

edited

Loading