Skip to content

Commit

Permalink
chore: edc 0.7.1 infrastructure update (eclipse-tractusx#351)
Browse files Browse the repository at this point in the history
Chore/edc 0.7.0 infra
  • Loading branch information
mhellmeier authored May 21, 2024
2 parents ffd76f0 + ec0b456 commit e1fb8e8
Show file tree
Hide file tree
Showing 25 changed files with 1,289 additions and 218 deletions.
1 change: 1 addition & 0 deletions .tractusx
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ repositories:
skipReleaseChecks:
alignedBaseImage:
- "local/vault/Dockerfile"
- "local/iam-mock/Dockerfile"
55 changes: 43 additions & 12 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Setup development database

For local development a postgresql database is needed. The file local/docker-compose-dev-postgres.yaml provides a
For local development a postgresql database is needed. The file local/docker-compose-dev-postgres.yaml provides a
postgres that can be started freshly for development:

```shell
Expand All @@ -16,19 +16,23 @@ docker compose -f docker-compose-dev-postgres.yaml down
_NOTE: For testing purposes HyperSql is still used but excluded for spring run._

## Keeping dependencies-files up to date

### Backend

Navigate to the `./backend` folder and run:
Navigate to the `./backend` folder and run:

```
mvn org.eclipse.dash:license-tool-plugin:license-check
cp DEPENDENCIES ../DEPENDENCIES_BACKEND
```
The first line runs the maven license tool with the parameters specified in the

The first line runs the maven license tool with the parameters specified in the
`./backend/pom.xml` and produces a DEPENDENCIES file in the .`/backend` folder.
Then this file gets copied to the PURIS-project root folder under the name `DEPENDENCIES_BACKEND`.
Both files should be updated prior to any pull request.
Then this file gets copied to the PURIS-project root folder under the name `DEPENDENCIES_BACKEND`.
Both files should be updated prior to any pull request.

### Frontend

```
# move to a persistent folder. Could also be ~/jars.
mv org.eclipse.dash.licenses-1.0.2.jar ~/coding/org.eclipse.dash.licenses-1.0.2.jar
Expand All @@ -42,22 +46,43 @@ cd frontend
eclipseDashTool package-lock.json -project automotive.tractusx -summary ../DEPENDENCIES_FRONTEND
```

### mock-util-service

Temprorary also the mock-util-service needs to be kept up to date. First add the alias same as for frontend to your
`.bashrc` and then use the following command:

```shell
cd local/iam-mock
cat requirements.txt | grep -v \# \
| sed -E -e 's|([^= ]+)==([^= ]+)|pypi/pypi/-/\1/\2|' -e 's| ||g' \
| sort | uniq \
| eclipseDashTool -summary DEPENDENCIES -
```

Note: Dash action provided by eclipse-tractusx/sig-infra does not provide to opportunity for python.

## Frontend container building workaround to use environment variables for vue

### The mechanism for docker is the following:

- `.env` has vite variables
- `.env.dockerbuild` has the vite variable that maps on an environment variable (`VITE_BACKEND_BASE_URL=$BACKEND_BASE_URL`)
- `.env.dockerbuild` has the vite variable that maps on an environment
variable (`VITE_BACKEND_BASE_URL=$BACKEND_BASE_URL`)
- `src/config.json` has the environment variable names and the environment variable to substring in a json format.

### When building the container:

1. `.env.dockerbuild` is used
2. vite / vue builds the application into a dest folder, that will be served by nginx

> Result for the .env: <br> VITE_BACKEND_BASE_URL won't write a variable value BUT a placeholder into the built files ($BACKEND_BASE_URL)
> Result for the .env: <br> VITE_BACKEND_BASE_URL won't write a variable value BUT a placeholder into the built files (
> $BACKEND_BASE_URL)
### When building the container, there is a "start-nginx.sh" file that does the following

1. Collects the environment variables (set for the docker container / set via helm as `BACKEND_BASE_URL`)
2. Looks-up the "to replace string" from `config.json` (e.g., for `BACKEND_BASE_URL`, it will search for `$BACKEND_BASE-URL` in the built files)
2. Looks-up the "to replace string" from `config.json` (e.g., for `BACKEND_BASE_URL`, it will search
for `$BACKEND_BASE-URL` in the built files)
3. Does the replacement in the built files
4. Starts nginx

Expand All @@ -66,26 +91,30 @@ eclipseDashTool package-lock.json -project automotive.tractusx -summary ../DEPEN
When changing the helm charts due to changes of e.g. environment variables, one should test locally whether the changes
work.

First thing one should check is whether the templates may be resolved / substituted correctly and if your changes are
First thing one should check is whether the templates may be resolved / substituted correctly and if your changes are
defaulted correctly:

```shell
cd charts/puris
helm template .
>> no error is thrown, chart is resolved, changes are done correctly
```

Now build your images as explained in the respective install.mds.

- [backend](../backend/INSTALL.md)
- [frontend](../frontend/INSTALL.md)

Now you need to update your Chart.yml and values.yml:

- Chart.yml: change `appVersion` to your build tag (e.g., `dev`)
- values.yml: for both frontend and backend change the image
- `repository` should be set to the image name used during docker build (e.g., `puris-backend`, `puris-frontend`)
- `pullPolicy` should be set to `Never`
- `repository` should be set to the image name used during docker build (e.g., `puris-backend`, `puris-frontend`)
- `pullPolicy` should be set to `Never`

Now depending on your runtime environment you need to load the images into it (we assume, you built puris-backend:dev
Now depending on your runtime environment you need to load the images into it (we assume, you built puris-backend:dev
and (puris-frontend:dev):

```shell
# minikube
minikube image load puris-backend:dev
Expand All @@ -96,6 +125,7 @@ docker image ls | grep puris-backend
minikube image ls --format table | grep puris-frontend
docker image ls | grep puris-frontend
```

```shell
# kind
kind load puris-backend:dev
Expand All @@ -108,6 +138,7 @@ docker image ls | grep puris-backend
docker exec -it {container-id} crictl images | grep puris-frontend
docker image ls | grep puris-frontend
```

**ATTENTION: MAKE SURE THAT THE IMAGE ID IN YOUR KUBERNETES ENVIRONMENT IS THE SAME AS IN YOUR LOCAL DOCKER.**

Else you can delete images as follows:
Expand Down
1 change: 1 addition & 0 deletions local/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
*.keys
*.secret
.env
seed-bdrs.sh
72 changes: 50 additions & 22 deletions local/INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,34 @@

## Additional information for Windows users

If you want to use the local setup on a Windows machine, it seems advisable to use the Windows subsystem for Linux (WSL).
An installation guide can be found [here](https://learn.microsoft.com/en-us/windows/wsl/install).
If you want to use the local setup on a Windows machine, it seems advisable to use the Windows subsystem for Linux (
WSL).
An installation guide can be found [here](https://learn.microsoft.com/en-us/windows/wsl/install).

Also see this [additional information about using Docker in combination with WSL2](https://docs.docker.com/desktop/wsl/)

## Initial Setup
In case you had any previous installations of this project on your machine, it is advisable to remove them via the script

In case you had any previous installations of this project on your machine, it is advisable to remove them via the
script
(see below in the Notes on debugging section).

Run the following script to generate the necessary keys. It will also create an .env file in the ./local folder.
Make sure to have `openssl` and `jq` installed in your shell. openssl is pre-installed on most operating systems. jq can be
Run the following script to generate the necessary keys. It will also create an .env file in the ./local folder.
Make sure to have `openssl` and `jq` installed in your shell. openssl is pre-installed on most operating systems. jq can
be
installed via the usual installation repositories, see [here](https://jqlang.github.io/jq/download/)

```shell
cd local
sh generate-keys.sh
```

## Build
If you are doing a fresh install and everytime you edited the code of the PURIS frontend or backend you have to create a
new build of docker images for the PURIS frontend/backend.

For creating a docker image of the frontend, navigate your shell to the frontend folder and run
If you are doing a fresh install and everytime you edited the code of the PURIS frontend or backend you have to create a
new build of docker images for the PURIS frontend/backend.

For creating a docker image of the frontend, navigate your shell to the frontend folder and run

```
docker build -t puris-frontend:dev .
Expand All @@ -37,50 +43,72 @@ docker build -t puris-backend:dev .

Please see the INSTALL.md documents in the [frontend](../frontend/INSTALL.md) and [backend](../backend/INSTALL.md)

The default image tag is 'dev'. Remember to also adjust the tag in the docker-compose.yaml if you want to use different
tags.
The default image tag is 'dev'. Remember to also adjust the tag in the docker-compose.yaml if you want to use different
tags.

## Start
First start the infrastructure by navigating your shell to the local folder and running

First start the infrastructure by navigating your shell to the local folder and running

```shell
docker compose -f docker-compose-infrastructure.yaml up
```
After the MIW container has finished booting, use this script (also in the local folder) to initialise two wallets for customer and supplier:

Note: sh init-wallets.sh is temporarily not needed
After the MIW container has finished booting, use this script (also in the local folder) to initialise two wallets for
customer and supplier:

```shell
sh init-wallets.sh
```
Then start the PURIS demonstrator containers via:

After starting the central infrastructure, initialize the bdrs-service. To do so, just run the script `seed-bdrs.sh`
created during the run of script `generate-keys.sh`.

```shell
sh seed-brds.sh
```

Then start the PURIS demonstrator containers via:

```shell
docker compose up
```
Wait for the startup and visit http://localhost:3000/ for the customer's frontend or http://localhost:3001/ for the supplier side.

Whenever you have edited the source code in the frontend or backend and you want to test these changes, we recommend that you
stop all the containers, which are not part of the infrastructure, by deleting the volumes, i.e. run
Wait for the startup and visit http://localhost:3000/ for the customer's frontend or http://localhost:3001/ for the
supplier side.

Whenever you have edited the source code in the frontend or backend and you want to test these changes, we recommend
that you
stop all the containers, which are not part of the infrastructure, by deleting the volumes, i.e. run

```
docker compose down -v
```

Then, build a new docker image for the respective component (frontend and/or backend) as described above and then restart via
Then, build a new docker image for the respective component (frontend and/or backend) as described above and then
restart via

```shell
docker compose up
```

In general, it is not necessary to restart the infrastructure, if you had to rebuild frontend or backend.
However, in rare cases there may be issues with the MIW. If this
happens, you should use the cleanup script as mentioned in the debugging section below and then repeat the above-mentioned
steps beginning with the Initial Setup section.
In general, it is not necessary to restart the infrastructure, if you had to rebuild frontend or backend.
However, in rare cases there may be issues with the MIW. If this
happens, you should use the cleanup script as mentioned in the debugging section below and then repeat the
above-mentioned
steps beginning with the Initial Setup section.

## Notes on debugging

### Vault & Certs

When having problems with the certs or the vault, one may need to delete the vault container.
The following script stops all infrastructure containers as well as the PURIS demonstrator containers:
The following script stops all infrastructure containers as well as the PURIS demonstrator containers:

```shell
cd local
sh cleanup.sh
```

Then start your containers again with the aforementioned commands.
9 changes: 9 additions & 0 deletions local/bdrs/application.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
edc.api.auth.key=${EDC_API_PW}
web.http.management.port=8581
web.http.management.path=/api/management
web.http.directory.port=8582
web.http.directory.path=/api/directory
# looking up DIDs should not use https
edc.iam.did.web.use.https=false
edc.iam.trusted-issuer.issuer.id=did:web:mock-util-service/trusted-issuer
_level=DEBUG
3 changes: 3 additions & 0 deletions local/cleanup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
docker compose down -v
docker compose -f docker-compose-infrastructure.yaml down -v
docker image rm local-vault
docker image rm local-mock-util-service
rm .env
rm seed-bdrs.sh
rm ./vault/secrets -r
rm ./iam-mock/keys -r
echo "Deleted .env and vault/secrets"
29 changes: 26 additions & 3 deletions local/docker-compose-infrastructure.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,13 @@ services:
image: postgres:15.4-alpine
container_name: postgres-miw
environment:
POSTGRES_DB: edc
POSTGRES_DB: miw
POSTGRES_USER: ${PG_USER}
POSTGRES_PASSWORD: ${PG_PW}
volumes:
# volumes:
# use docker-compose down --volumes to kill db volume
# only then, changes to the script are executed!
- ./postgres/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
# - ./postgres/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
ports:
- "127.0.0.1:5432:5432"
networks:
Expand Down Expand Up @@ -81,6 +81,29 @@ services:
- ./vault/secrets:${VAULT_SECRETS_DIR}
networks:
- miw-net

mock-util-service:
build: ./iam-mock
container_name: mock-util-service
ports:
- "127.0.0.1:8888:80"
networks:
- miw-net

bdrs:

Check warning on line 93 in local/docker-compose-infrastructure.yaml

View workflow job for this annotation

GitHub Actions / Analyze

[MEDIUM] Healthcheck Not Set

Check containers periodically to see if they are running properly.
image: tractusx/bdrs-server-memory:0.0.4
container_name: bdrs
env_file:
- ./bdrs/application.properties
environment:
WEB_HTTP_PORT: 8580
WEB_HTTP_PATH: /api
ports:
- "127.0.0.1:8580:8580"
- "127.0.0.1:8581:8581"
- "127.0.0.1:8582:8582"
networks:
- miw-net
networks:
miw-net:
name: miw-net
Loading

0 comments on commit e1fb8e8

Please sign in to comment.