Investigate EDC0 for known MS8 issues

[richardtreier]

Feature Request

Description

There's a few things we waited for EDC 0.1.0 to either validate as fixed, (re-)create as issues or implement workarounds for.

Work Breakdown

• depends on EDC 0.2.x Migration in edc-extensions #391
• Check our EDC with Core EDC >0 for the following issues:
  • PosgreSQL error when querying >5k results.
  • N+1 SELECT problem affecting transfer processes (slowdown of >10s / 50 data offers)
    • Seems to be fixed for now. A connector with data offers 1, 50, 80 and 500 all average about 3-4s for transfer processes.
  • N+1 SELECT problem affecting catalog (slowdown of >10s / 50 data offers)
    • Seems to persist. 500 assets take 12s on local machine without DB latency.
    • Core EDC has positioned themselves against this issues in favor of Web-DIDs and the Federated Catalog which would periodically crawl ALL connectors in a data space.
  • No clear quantity structure supported by EDC. EDC is tested with in-memory stores. Moved into initiative https://github.com/sovity/PMO-Software/issues/536
  • Contract Negotiation does not persist asset information at time of negotiation agreement. Moved into issue Consuming Contract Agreements contain no Asset Information #336
  • Possibly breaking changes in persistence that cannot be dealt with, or cannot be easily dealt with with database migrations.
  • Validate Connector ID concept for consistency (see Management API Transfer History endpoint, for example)
  • The MS8 Connector's Contract Negotiations fail in HTTPS-only environments. See EDC Connector Configuration and Contract Negotiation #379
  • Transfer Processes don't get put into the state COMPLETED on the providing side.
  • Investigate an old issue where the number of contract offers grew with numAssets to the power of numContractDefinitions.
  • Data consistency issues. Moved into issue Prevent Broken States by Deletion / Editing edc-ui#462
  • Check EDC 0 Capabilities for tracing (?) logging. Logging has been improved with configurable dev logging. Tracing is used in the EE, but not the CE as of now.
• Create Issues to either ignore, fix or workaround the above issues.

[richardtreier]

Findings of @efiege on a local machine (no latency to db):

2 Assets:
Catalog retrieve took milliseconds: 525
Transfer took milliseconds: 2525

51 Assets:
Catalog retrieve took milliseconds: 5855
Transfer took milliseconds: 3013

81 Assets:
Catalog retrieve took milliseconds: 3279
Transfer took milliseconds: 4203

501 Assets:
Catalog retrieve took milliseconds: 12975
Transfer took milliseconds: 3638

[dcarmen1]

Hi!

I wanted to ask about the status of the issue [387]. I understand that you're likely handling a multitude of tasks, but I'm curious if there have been any updates or progress made regarding the request.

Thank you for the help.

[richardtreier]

I wanted to ask about the status of the issue [387]. I understand that you're likely handling a multitude of tasks, but I'm curious if there have been any updates or progress made regarding the request.

Hi, our migration to EDC 0.x is still underway, albeit at a very high priority.

At this point we aren't ready yet to test all of the above points as we are still migrating core functionalities.

If you want to track our progress on our EDC 0.x migration, feel free to check out our issues:

• https://github.com/sovity/edc-extensions/milestone/1
• EDC 0.2.x Migration in UI + API Wrapper edc-ui#364

[dcarmen1]

Hi, regarding the status of issue [387] I'm eager to know if there have been any updates or progress made concerning the request, is the implementation inside of the new connector version?
Thank you

[tmberthold]

Hi, regarding the status of issue [387] I'm eager to know if there have been any updates or progress made concerning the request, is the implementation inside of the new connector version? Thank you

Are you looking for a specific point in this issue? We have migrated to core-edc 0.2.1 in the meanwhile, for example this core-edc version is used in out last CE: https://github.com/sovity/edc-extensions/releases/tag/v7.1.1

[dcarmen1]

Yes I was looking for this point: The MS8 Connector's Contract Negotiations fail in HTTPS-only environments

[richardtreier]

Issues have been split where appliccable or marked as completed.