Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgraded RML & added logic to handle new key in metadata.json #526

Merged
merged 2 commits into from
Mar 21, 2024
Merged

upgraded RML & added logic to handle new key in metadata.json #526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
18873c9
upgraded RML & added logic to handle new key in metadata.json
Dragos0000 Mar 21, 2024
8fba211
upgraded RML & added logic to handle new key in metadata.json
Dragos0000 Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ stop-metabase:
init-rml-mapper:
@ echo -e "RMLMapper folder initialisation!"
@ mkdir -p ./.rmlmapper
@ wget -c https://github.com/RMLio/rmlmapper-java/releases/download/v6.1.3/rmlmapper-6.1.3-r367-all.jar -O ./.rmlmapper/rmlmapper.jar
@ wget -c https://github.com/RMLio/rmlmapper-java/releases/download/v6.2.2/rmlmapper-6.2.2-r371-all.jar -O ./.rmlmapper/rmlmapper.jar

init-limes:
@ echo -e "Limes folder initialisation!"
Expand Down
3 changes: 2 additions & 1 deletion infra/airflow-cluster/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ FROM docker.io/apache/airflow:2.5.1-python3.8

# quick sudo
USER root
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A8D3785C
RUN apt-get update && apt-get install make && apt-get install build-essential git -y && apt-get install -y wget && apt-get install -y unzip
RUN apt install -y default-jre

Expand All @@ -14,7 +15,7 @@ COPY requirements.txt /opt/airflow
# working in the /opt/airflow
WORKDIR /opt/airflow
RUN mkdir -p ./.rmlmapper
RUN wget -c https://github.com/RMLio/rmlmapper-java/releases/download/v6.1.3/rmlmapper-6.1.3-r367-all.jar -O ./.rmlmapper/rmlmapper.jar
RUN wget -c https://github.com/RMLio/rmlmapper-java/releases/download/v6.2.2/rmlmapper-6.2.2-r371-all.jar -O ./.rmlmapper/rmlmapper.jar


RUN wget -c https://kumisystems.dl.sourceforge.net/project/saxon/Saxon-HE/10/Java/SaxonHE10-6J.zip -P .saxon/
Expand Down
1 change: 1 addition & 0 deletions infra/airflow/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ FROM docker.io/apache/airflow:2.5.1-python3.8

# quick sudo
USER root
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A8D3785C
RUN apt-get update && apt-get install make && apt-get install build-essential git -y && apt-get install -y wget && apt-get install -y unzip
RUN apt install -y default-jre

Expand Down
2 changes: 2 additions & 0 deletions ted_sws/data_manager/adapters/mapping_suite_repository.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,8 @@ def _read_mapping_suite_package(self, mapping_suite_identifier: str) -> Optional
package_path = self.repository_path / mapping_suite_identifier
if package_path.is_dir():
package_metadata = self._read_package_metadata(package_path)
package_metadata["version"] = package_metadata[
"version"] if "version" in package_metadata else package_metadata["mapping_version"]
package_metadata["identifier"] = package_metadata[
"identifier"] if "identifier" in package_metadata else mapping_suite_identifier
package_metadata["transformation_rule_set"] = self._read_transformation_rule_set(package_path)
Expand Down
7 changes: 5 additions & 2 deletions ted_sws/mapping_suite_processor/services/mapping_suite_validation_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
from ted_sws.mapping_suite_processor.adapters.mapping_suite_structure_checker import MappingSuiteStructureValidator

MAPPING_SUITE_METADATA_IDENTIFIER_KEY = 'identifier'
MAPPING_SUITE_METADATA_VERSION_KEY = 'version'
STANDARD_MAPPING_SUITE_METADATA_VERSION_KEY = 'version'
EFORMS_MAPPING_SUITE_METADATA_VERSION_KEY = 'mapping_version'


def get_mapping_suite_id_from_file_system(mapping_suite_path: pathlib.Path) -> Optional[str]:
Expand All @@ -20,7 +21,9 @@ def get_mapping_suite_id_from_file_system(mapping_suite_path: pathlib.Path) -> O
if mapping_suite_metadata_path.exists() and mapping_suite_metadata_path.is_file():
mapping_suite_metadata = json.loads(mapping_suite_metadata_path.read_text(encoding="utf-8"))
identifier_value = mapping_suite_metadata[MAPPING_SUITE_METADATA_IDENTIFIER_KEY]
version_value = mapping_suite_metadata[MAPPING_SUITE_METADATA_VERSION_KEY]
version_value = mapping_suite_metadata[
STANDARD_MAPPING_SUITE_METADATA_VERSION_KEY] if STANDARD_MAPPING_SUITE_METADATA_VERSION_KEY in mapping_suite_metadata else \
mapping_suite_metadata[EFORMS_MAPPING_SUITE_METADATA_VERSION_KEY]
return f"{identifier_value}_v{version_value}"
return None

Expand Down
1 change: 1 addition & 0 deletions tests/test_data/notice_transformer/test_repository/test_package4/metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"mapping_version": "3.0.0-alpha.1",
"ontology_version": "4.0.0",
"mapping_type": "eforms",
"mapping_suite_hash_digest": "1740b31999a492bbe80122f766f8d4c1dd7a3f720b95337a88a10cf673ceb223",
"metadata_constraints": {
"constraints": {
"eforms_subtype": [
Expand Down
Loading