metadata

Amundsen Metadata Service

Amundsen Metadata service serves Restful API and is responsible for providing and also updating metadata, such as table & column description, and tags. Metadata service can use Neo4j, Apache Atlas, AWS Neptune Or Mysql RDS as a persistent layer.

For information about Amundsen and our other services, refer to this README.md. Please also see our instructions for a quick start setup of Amundsen with dummy data, and an overview of the architecture.

Requirements

• Python >= 3.8

Doc

• https://www.amundsen.io/amundsen/

Instructions to start the Metadata service from distribution

$ venv_path=[path_for_virtual_environment]
$ python3 -m venv $venv_path
$ source $venv_path/bin/activate
$ pip3 install amundsen-metadata
$ python3 metadata_service/metadata_wsgi.py

-- In a different terminal, verify getting HTTP/1.0 200 OK
$ curl -v http://localhost:5002/healthcheck

Instructions to start the Metadata service from the source

$ git clone https://github.com/amundsen-io/amundsenmetadatalibrary.git
$ cd amundsenmetadatalibrary
$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install -e ".[all]" .
$ python3 metadata_service/metadata_wsgi.py

-- In a different terminal, verify getting HTTP/1.0 200 OK
$ curl -v http://localhost:5002/healthcheck

Instructions to start the service from Docker

$ docker pull amundsendev/amundsen-metadata:latest
$ docker run -p 5002:5002 amundsendev/amundsen-metadata
# - alternative, for production environment with Gunicorn (see its homepage link below)
$ ## docker run -p 5002:5002 amundsendev/amundsen-metadata gunicorn --bind 0.0.0.0:5002 metadata_service.metadata_wsgi

-- In a different terminal, verify getting HTTP/1.0 200 OK
$ curl -v http://localhost:5002/healthcheck

Production environment

By default, Flask comes with Werkzeug webserver, which is for development. For production environment use production grade web server such as Gunicorn.

$ pip install gunicorn
$ gunicorn metadata_service.metadata_wsgi

Here is documentation of gunicorn configuration.

Configuration outside local environment

By default, Metadata service uses LocalConfig that looks for Neo4j running in localhost. In order to use different end point, you need to create Config suitable for your use case. Once config class has been created, it can be referenced by environment variable: METADATA_SVC_CONFIG_MODULE_CLASS

For example, in order to have different config for production, you can inherit Config class, create Production config and passing production config class into environment variable. Let's say class name is ProdConfig and it's in metadata_service.config module. then you can set as below:

METADATA_SVC_CONFIG_MODULE_CLASS=metadata_service.config.ProdConfig

This way Metadata service will use production config in production environment. For more information on how the configuration is being loaded and used, here's reference from Flask doc.

Apache Atlas

Amundsen Metadata service can use Apache Atlas as a backend. Some of the benefits of using Apache Atlas instead of Neo4j is that Apache Atlas offers plugins to several services (e.g. Apache Hive, Apache Spark) that allow for push based updates. It also allows to set policies on what metadata is accessible and editable by means of Apache Ranger.

Apache Atlas is a data governance service meaning you also get Apache Atlas UI for easy access to your metadata. It is, however, aimed for administrators rather then end-users, which we suggest to direct towards Amundsen UI.

Apache Atlas is so far the only proxy in Amundsen supporting both push and pull for collecting metadata:

• Push method by leveraging Apache Atlas Hive Hook. It's an event listener running alongside Hive Metastore, translating Hive Metastore events into Apache Atlas entities and pushing them to Kafka topic, from which Apache Atlas ingests the data by internal processes.
• Pull method by leveraging Amundsen Databuilder integration with Apache Atlas. It means that extractors available in Databuilder can be used to collect metadata about external systems (like PostgresMetadataExtractor) and sending them to Apache Atlas in a shape consumable by Amundsen.

If you would like to use Apache Atlas as a backend for Metadata service you will need to create a Config as mentioned above. Make sure to include the following:

PROXY_CLIENT = PROXY_CLIENTS['ATLAS'] # or env PROXY_CLIENT='ATLAS'
PROXY_PORT = 21000          # or env PROXY_PORT
PROXY_USER = 'atlasuser'    # or env CREDENTIALS_PROXY_USER
PROXY_PASSWORD = 'password' # or env CREDENTIALS_PROXY_PASSWORD

To start the service with Atlas from Docker. Make sure you have atlasserver configured in DNS (or docker-compose)

$ docker run -p 5002:5002 --env PROXY_CLIENT=ATLAS --env PROXY_PORT=21000 --env PROXY_HOST=atlasserver --env CREDENTIALS_PROXY_USER=atlasuser --env CREDENTIALS_PROXY_PASSWORD=password amundsen-metadata:latest

---

NOTE

While Apache Atlas supports fine grained access, Amundsen does not support this yet.

Developer guide

Code style

• PEP 8: Amundsen Metadata service follows PEP8 - Style Guide for Python Code.
• Typing hints: Amundsen Metadata service also utilizes Typing hint for better readability.

API documentation

We have Swagger documentation setup with OpenApi 3.0.2. This documentation is generated via Flasgger. When adding or updating an API please make sure to update the documentation. To see the documentation run the application locally and go to localhost:5002/apidocs/. Currently the documentation only works with local configuration.

Code structure

Please visit Code Structure to read how different modules are structured in Amundsen Metadata service.

Roundtrip tests

Roundtrip tests are a new feature - by implementing the abstract_proxy_tests and some test setup endpoints in the base_proxy, you can validate your proxy code against the actual data store. These tests do not run by default, but can be run by passing the --roundtrip-[proxy] argument. Note this requires a fully-configured backend to test against.

$ python -m pytest --roundtrip-neptune .