A collection of tools for Data Mesh architectural approach.
The library can be used to validate your .yaml data contract against Open Data Contract Standard JSON schema from Bitol-io / open-data-contract-standard
Install it from PyPi
pip install -U datamesh
Install latest nightly builds from git
pip install git+https://github.com/georgegach/DataMesh.git
data-contract-validation <path_to_contract_yaml_file> <optional_path_to_standards_json_schema>or in Python
from datamesh.contract import Validator
validation_errors = (
Validator(
contract="examples/all/postgresql-adventureworks-contract.yaml",
standard="schema/odcs-json-schema.json" # This is optional
)
.print_report() # Prints validation results and returns `self`
.errors # List of errors
)🚩 Validation errors:
.
│
├──[]
│ └── 'datasetName' is a required property
│
├──[]
│ └── 'quantumName' is a required property
│
├──['kind']
│ └── 'managedDataset' is not one of ['DataContract']
can be used within your CI as
...
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install datamesh
- name: Run Data Contract Validation on all examples
run: |
set -o pipefail
failed_files=""
for file in examples/all/*.yaml; do
echo "Validating $file..."
if ! data-contract-validation "$file"; then
echo "Validation failed for $file"
failed_files="$failed_files $file"
fi
done
if [ -n "$failed_files" ]; then
echo "Validation failed for the following files:$failed_files"
exit 1
else
echo "All files validated successfully."
fi
...Streamlit App at datamesh.streamlit.app
Feel free to contribute to the project under free and open-source GPLv3 license.