simple python pipeline engine that generates visualization for execution timeline information and steps dependencies in a graph
https://networkgraphs.github.io/pipeline-visualization/
The pipeline can bun run with
run
which will execute runner.py
from venv
it takes as input a manifest file, as example
# Stage > Job
pipeline:
fetch:
fetch-data-1: pipeline.py#fetch
process:
calculate-functions: pipeline.py#calculate
compute-statistics: pipeline.py#compute
build:
generate-website: pipeline.py#build
which is a map of stages and jobs ponting to python files and entry point functions from pipeline.py
The execution will generate
- pipeline.json : a list of all execution steps and their timing information
- artifacts.json : a list of all generated artifacts
- depedencies.json : a graph of depedencies between steps through artifacts usage
see example output in https://networkgraphs.github.io/pipeline-visualization/
testing containers e.g. graphviz rendering
create cache/test/example.dot
with content such as
digraph G {
A -> B;
B -> C;
C -> A;
}
note cache is mapped in /data
inside the container.
Then generate the graph with
run graphviz test/example.dot
this command will serve the content of ./docs
directory
cmd server