Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add/update documentation for connector kit #138

Merged
merged 12 commits into from
Apr 21, 2023
6 changes: 6 additions & 0 deletions docs/development/Release.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,11 @@ The Eclipse Bot is able to approve dependencies automatically, if the license ca
from maven central.
2. Create the Eclipse IP Issues or ask an Eclipse Commiter to do this for you.

## 4. Update OpenAPI docs

As part of the [kits documentation provided for docusaurus](../kit/development-view/page00_development_view.md) we provide an OpenAPI reference.
This refers to the [EDC API](https://github.com/eclipse-edc/Connector/tree/main/resources/openapi) and needs to be updated to the current release.
The yaml files found there are then converted with the [docusaurus openapi plugin](https://www.npmjs.com/package/docusaurus-plugin-openapi-docs).

bcronin90 marked this conversation as resolved.
Show resolved Hide resolved
[maven-shield]: https://img.shields.io/badge/Apache%20Maven-URL-blue
[maven-url]: https://maven.apache.org
Binary file added docs/kit/adoption-view/images/domain-model.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/kit/adoption-view/images/edc_overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 48 additions & 0 deletions docs/kit/adoption-view/page_adoption-view.md
bcronin90 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
id: Adoption View
title: Adoption View
description: 'Connector Kit'
sidebar_position: 1
---

The ConnectorKit provides a connector framework, based on the [Eclipse Dataspace Connector][edc-url] for sovereign, cross-enterprise data exchange.

![EDC Overview](images/edc_overview.png)

Trust, interoperability and data sovereignty, these are the objectives and values for secure and sustainable peer-to-peer data exchange between organizations and companies. The claim is data sovereignty: Whoever makes data available retains control and decides individually who is involved in the data exchange, how, when, where and under what conditions.

A corresponding concept was developed in the context of [Gaia-X][gaiax-url] and the [International Data Space Association][idsa-url]. The essential software component is the connector.

With the [EDC][edc-url], a new central communication component was created, which implements the following architectural principles:

- Simple, maintaining a small and efficient core with as few external dependencies as possible
- Interoperable, independent of platforms and ecosystems
- Decentralized, software components with the necessary capabilities for participating in a data room are located on the partners' side, data is only exchanged between the agreed points.
- Data protection is more important than data sharing, data to be transmitted are fundamentally linked to policies via contracts; a transfer without a contract is not possible.
- Separation of metadata and data enables high throughput rates for the actual data transfer.
- Consistent semantics for the data is the basis for the consistency of digital value creation.
- As far as possible, all processes, starting with determining the identity, through ensuring the contractually agreed regulations to data transmission, are automated.
- Existing standards and protocols ([GAIA-X][gaiax-url] and [IDSA][idsa-url]) are used as far as possible.

The [EDC][edc-url] as a connector implements a framework agreement for sovereign, cross-organizational data exchange. The International Data Spaces Standard (IDS) and relevant principles in connection with [GAIA-X][gaiax-url] were implemented. The connector is designed to be extensible to support alternative protocols and to be integrated into different ecosystems.

The objective is to set up a decentralized software component on the part of the respective partner, which bundles the skills required to participate in a data room and enables peer-to-peer connections between participants.
The focus here is particularly on the data sovereignty of the independent companies.
The functionality required for this is bundled in the open-source project "Eclipse Dataspace Connectors", to which the Catena-X partners contribute as part of the Eclipse Foundation.

The main difference between the EDC and the previous connectors of the [IDSA][idsa-url] is the separation of the communication into a channel for the metadata and one for the actual data exchange. The channel for the data supports various transmission protocols via so-called data plane extensions. The metadata is transmitted directly via the EDC interface, while the actual data exchange then takes place via the appropriate channel extension. In this way, a highly scalable data exchange is made possible.

![EDC Architecture](images/edc_architecture.png)

The architecture of the EDC combines various services that are necessary for the above principles:

- An interface to the Identity Provider service, currently [IDSA][idsa-url]'s [Dynamic Attribute Provisioning System][daps-url]. This central service provides the identity and the corresponding authentication of the participants in the data exchange. (There is no authorization at this point). Decentralized solutions will also be supported in the future.
- The provision of possible offers (contract offering) which, on the one hand, stipulates the data offered and the associated terms of use (policies) in corresponding contracts.
- An interface for manual selection of data and associated contract offers.
- The actual data transfer via the data plane extension
- The connection of software systems on the customer and provider side

stephanbcbauer marked this conversation as resolved.
Show resolved Hide resolved
[edc-url]: https://github.com/eclipse-edc/Connector
[gaiax-url]: https://www.data-infrastructure.eu/GAIAX/Navigation/EN/Home/home.html
[idsa-url]: https://internationaldataspaces.org/
[daps-url]: https://www.dataspaces.fraunhofer.de/en/software/identity_provider.html
64 changes: 64 additions & 0 deletions docs/kit/adoption-view/page_domain_model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
id: Domain Model
title: Domain Model
description: 'Connector Kit'
sidebar_position: 2
---

![domain-model](images/domain-model.png)

> The shown picture illustrates only a generic view of the Domain Model and is not intended to show all aspects of the project.

## Asset

An asset represents data (databases, files, cache information, etc.) which should be published and shared between
organizations. For each asset, a [`DataAddress`](#data-address) needs to be resolvable.

## Data address

A data address is a pointer into the physical storage location where an asset will be stored.

## Contract

A contract always contains one or more [`Assets`](#asset) and a single [`Policy`](#policy). The contract construct is
used to define the arrangement between two parties ("consumer" and "provider"). Regarding this arrangement, the contract
passes several stages which are explained below:

### Contract definition

Contract definitions associate a policy with assets. A `ContractDefinition` object contains an access policy, a contract
policy, and an asset selector which links the contract to one or more assets.

### Contract offer

The contract offer is a dynamic representation of the [`ContractDefinition`](#contract-definition)
for a specific consumer and serves as protocol's data transfer object (DTO) for a particular contract negotiation.
Contract offers are not persisted and will be regenerated on every request. The connector acting as data provider will
generate contract offers only for contract definitions dedicated to the organization or data space participant
operating the requesting connector acting as data consumer. A contract offer is always related to a single asset of
the `ContractDefinition` object (e.g. for a `ContractDefinition` containing three `Asset` objects, the connector will
generate three `ContractOffer` objects).

### Contract negotiation

A `ContractNegotiation` captures the current state of the negotiation of a contract (`ContractOffer` ->
`ContractAgreement`) between two parties. This process is inherently asynchronous.

### Contract agreement

A contract agreement represents the agreed-upon terms of access and usage of an asset's data between two data space
participants, including a start and an end date and further relevant information.

## Policy

Contract policies represent permitted and prohibited actions over a certain asset. These actions can be limited further
by constraints (temporal or spatial) and duties ("e.g. deletion of the data after 30 days").
florianrusch-zf marked this conversation as resolved.
Show resolved Hide resolved

## Transfer process

After a successful contract negotiation, a `DataRequest` is sent from a consumer connector to a provider connector to
initiate the data transfer. It references the requested [`Asset`](#asset) and [`ContractAgreement`](#contract-agreement)
as well as information about the [data destination](#data-address).

Similar to the `ContractNegotiation`, this object captures the current state of a data transfer. This process is
inherently asynchronous, so the `TransferProcess` objects are stored in a backing data store (`TransferProcessStore`).
24 changes: 24 additions & 0 deletions docs/kit/development-view/page00_development_view.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Development View

## Project Overview

TractusX is an initiative of companies under the umbrella of the Eclipse Foundation.
It is a pilot for the larger initiative of CatenaX.
A broader overview of the project can be found on the initiative's [Github page][tractusx-edc-link]
or the homepage of the [Eclipse Foundation](https://projects.eclipse.org/projects/automotive.tractusx).

## The EDC

The Eclipse Dataspace Connector is one of the core components facilitating TractusX.

:::note TractusX EDC or Core EDC?

This documentation is for TractusX EDC.
It includes the Core EDC with all of its functionality.
However, this core is supplemented by extensions that allow for the use of additional backends and connection types.
Furthermore, the provided Helm charts, build configuration and tests allow for a smoother deployment.
:::

You can find the repository for the TractusX EDC [here][tractusx-edc-link].

[tractusx-edc-link]: https://github.com/eclipse-tractusx/tractusx-edc
35 changes: 35 additions & 0 deletions docs/kit/development-view/page01_eclipse_foundation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# The Eclipse Foundation

## Eclipse Development Process

This Eclipse Foundation open project is governed by the Eclipse Foundation
Development Process and operates under the terms of the Eclipse IP Policy.

* <https://eclipse.org/projects/dev_process>
* <https://www.eclipse.org/org/documents/Eclipse_IP_Policy.pdf>

## Eclipse Contributor Agreement

In order to be able to contribute to Eclipse Foundation projects you must
electronically sign the Eclipse Contributor Agreement (ECA).

* <http://www.eclipse.org/legal/ECA.php>

The ECA provides the Eclipse Foundation with a permanent record that you agree
that each of your contributions will comply with the commitments documented in
the Developer Certificate of Origin (DCO). Having an ECA on file associated with
the email address matching the "Author" field of your contribution's Git commits
fulfills the DCO's requirement that you sign-off on your contributions.

For more information, please see the Eclipse Committer Handbook:
<https://www.eclipse.org/projects/handbook/#resources-commit>

## License

Code in TractusX EDC is published under the [Apache License](https://github.com/eclipse-tractusx/tractusx-edc/blob/main/LICENSE).

## Contact

Contact the project developers via the project's "dev" list.

* <https://accounts.eclipse.org/mailing-list/tractusx-dev>
26 changes: 26 additions & 0 deletions docs/kit/development-view/page02_repository_structure.md
florianrusch-zf marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Repository Structure

The repository for TractusX EDC can be found [here](https://github.com/eclipse-tractusx/tractusx-edc).
It contains the following components:

## EDC Extensions

The core EDC is extensible by design.
TractusX EDC provides such extensions.
These extensions and their documentation are available
[here](https://github.com/eclipse-tractusx/tractusx-edc/blob/main/edc-extensions/README.md).

## Gradle Files for EDC Builds

Builds of TractusX EDC are performed via Gradle.
To allow for different configurations, different builds are provided.
For example separate secrets backends are supported, but require separate builds of EDC.
Therefor, different builds are available for both
[data plane](https://github.com/eclipse-tractusx/tractusx-edc/blob/main/edc-dataplane/README.md)
and [control plane](https://github.com/eclipse-tractusx/tractusx-edc/blob/main/edc-controlplane/README.md),

## Helm Charts for EDC Deployment

To facilitate deployment of these different builds and their prerequisites,
Helm charts are provided. The charts and their documentation can be found
[here](https://github.com/eclipse-tractusx/tractusx-edc/blob/main/charts/README.md).
21 changes: 21 additions & 0 deletions docs/kit/development-view/page03_project_structure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Project Structure

## Issue Tracking

Issues are maintained in [GitHub Issues](https://github.com/eclipse-tractusx/tractusx-edc/issues).

## Reporting Vulnerabilities

Vulnerabilities in the TractusX code base are best reported directly to the
[Eclipse Foundation](https://www.eclipse.org/security/).

## Git Flow

The TractusX EDC repository uses a Git Flow, with `main` as the development branch and `releases` as the release branch.
Other branches should follow the naming conventions of `feature/x` or `hotfix/x`, though this is not strictly enforced.

## Tooling

We use Java 11 with Gradle for dependencies and builds.
We use [Spotless](https://github.com/diffplug/spotless) for code formatting.
florianrusch-zf marked this conversation as resolved.
Show resolved Hide resolved
Releases are in the form of Docker containers and Helm charts.
24 changes: 24 additions & 0 deletions docs/kit/operation-view/page00_operation_view.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Software Operation View

## Introduction

The following documentation will guide you through the TractusX EDC deployment.
You will be setting up multiple controllers and enabling communication between them.

:::note TractusX EDC or Core EDC?

The following guide assumes the use of the TractusX EDC.
It includes the Core EDC with all of its functionality.
However, this core is supplemented by extensions that allow for the use of additional backends and connection types.
Furthermore, the provided Helm charts, build configuration and tests allow for a smoother deployment.
:::

## Connector Components

In a usual EDC environment, each participant would operate at least one connector.
Each of these connectors consists of a control plane and a data plane.
The control plane functions as administration layer and is responsible for resource management, contract negotiation and administering data transfer.
The data plane does the heavy lifting of transferring and receiving data streams.

Each of these planes comes in several variants, allowing for example secrets to be stored in Azure Vault or a Hashicorp Vault.
The setup on the following pages assumes the use of Hashicorp Vault for secrets and PostgreSQL for data storage.
43 changes: 43 additions & 0 deletions docs/kit/operation-view/page02_technical_prerequisites.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Technical Prerequisites

## Obtaining Releases

The most recent release of TractusX EDC can be obtained under `https://github.com/eclipse-tractusx/tractusx-edc/releases`.
To create your own build, you can clone the repository at `https://github.com/eclipse-tractusx/tractusx-edc` and consult the provided README.md.
This can be useful if you want to use non-standard extensions or configuration.

## Container Environment

TractusX releases come in the form of Docker containers and corresponding Helm charts.
As such, recent versions of the following are required.

- Docker
- Kubernetes
- Helm

Seeing as these are standard tools, TractusX EDC will run on any cloud environment that can accept Helm charts.

## Backend Dependencies

The EDC requires backend services for persistence of data and secrets. The following backends are currently supported.

Data Storage:

- PostgreSQL database
- In memory database

Secret Storage:

- Hashicorp Vault
- Azure Vault

The default setup assumes data storage via PostgreSQL database.
In memory storage is only recommended for running tests.
Hashicorp Vault is the default secret provider, because it is platform-agnostic.

Helm charts are provided to set up these services locally.
**These are not suited for production environments.**

## All-in-one deployment

An all-in-one deployment is no longer in scope and will not be provided.
Loading