Ideas about extended tests

[adur1990]

Hello everybody!

What is the problem?

So, as you can see in some PRs (#39, #40, #41, #42, #43) and off-line discussions, we encountered a problem with dtn7-go, as it starts to fail in larger scenarios, which we tried to fix without success.

How to deal with it?

The idea came up to build a large test setup, where the issue happens reproducibly.
I wanted to start a discussion thread here to get things done.

What have I done so far?

In the past couple days I built a docker container, that spawns a CORE sessions, which in turn starts 30 virtual nodes that are connected in a more or less random fashion (see attached picture).

Using environment variables, each node starts to sent BUNDLES_PER_NODE bundles with size PAYLOAD_SIZE bytes over a time period of EXPERIMENT_RUNTIME seconds, where the variables are configurable using mentioned environment variables.
Afterwards, all logs are collected and stored in a TAR archive for further analysis.

This works using the CORE GUI for development and debugging, but also works using the CORE Python API for automated execution.

How to proceed?

This is why I started this discussion.
How do you (as the community) think this should be incorporated?
My idea was to create a repo for all the CORE and Docker related stuff and then to have a Github Action in the main dtn7 repo (this one), where the docker container can be executed automatically.
But maybe there are better ways to do this?

[oxzi]

First, thanks a lot for your efforts. I am glad that this issue is actively being pursued.

In the past couple days I built a docker container, that spawns a [CORE](https://github.com/coreemu/core) sessions, which in turn starts 30 virtual nodes that are connected in a more or less random fashion (see attached picture).

Would you mind sharing your code, e.g., within a new branch in a subdirectory of contrib? For reference, I would still like to point to dtn7-playgorund [0].

My idea was to create a repo for all the CORE and Docker related stuff and then to have a Github Action in the main dtn7 repo (this one), where the docker container can be executed automatically.

As for the repo, feel free to continue using dtn7-playground or create a new one, I'll leave that up to you. If I am not mistaking, we can run jobs for up to six hours [1]. IIRC, there was also a maximum run time per month, but I did not find a reference in a hurry. Currently, there is already a nightly GitHub Action [2] to execute tests. I would love to see this expanded accordingly. Here I am happy to help - feel free to drop me a line. [0] https://github.com/dtn7/dtn7-playground [1] https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration [2] https://github.com/dtn7/dtn7-go/blob/master/.github/workflows/nightly.yml

[adur1990]

Awesome, was not aware of the playground.
So, I'm going to make a new branch there and upload and document my stuff there.
In this repo, I will create a new temporary Github Action to test and debug everything and as soon as everything is working as I expect it to be I will post an update here, ok?

[adur1990]

Alright, Jonas and I built something we can talk about, I guess.

The dtn7-playground repo now has a new branch, actions, which runs Github Actions with defined tests.
We built the whole thing so that it installs CORE and you can define arbitrary actions.
To do this, make an xml-based scenario with associated CORE Python script and a workflow YAML that defines the tests.
Currently there is an example, which is also the first test case that was the reason that we started the whole thing here in the first place.
30 virtual nodes are started that are sending payloads to each other.
How long the test runs, how many and how big the payloads are is defined via environment variables with a matrix in the workflow definition.
If a deadlock occurs and no more bundles can be sent, the test is marked as an error.
At the end, all logs are collected and made available as artifacts.

The question now is how to proceed.
I don't think it makes sense to push all the stuff into the dtn7-go main repo.
It is well kept in the playground.
But how should the workflow be triggered?
Just nightly?
Makes no sense to run an arbitrary number of tests every night, which will always have the same result, does it?
We could use the [Repository Dispatch]/https://docs.github.com/en/actions/reference/events-that-trigger-workflows#repository_dispatch) event to start the workflow in the dtn7-playground on pushes or pull requests in the dtn7-go main repo.

If that's ok I would open a PR with a customized workflow here in the main repo.

[adur1990]

@oxzi

[oxzi]

Sorry for the delayed reply. A few things have fallen through for me. So far I have only read over the changes, however I like what I read. Maybe you could create the logs as JSON, which would make the evaluation easier.

The question now is how to proceed. […] We could use the [Repository Dispatch]/https://docs.github.com/en/actions/reference/events-that-trigger-workflows#repository_dispatch) event to start the workflow in the dtn7-playground on pushes or pull requests in the dtn7-go main repo.

That's a good idea. Currently the same logic is already present in dtn7-go to automatically build a Nix package for both unstable as well as stable versions. I guess you could kind of copy the update-nur-packages.yml [0].

If that's ok I would open a PR with a customized workflow here in the main repo.

Please, I would be very happy about that. [0] https://github.com/dtn7/dtn7-go/blob/master/.github/workflows/update-nur-packages.yml

[adur1990]

So, now that the PRs are merged and the tests are working, how should be proceed?
Do you need any further information about the issue?
Any further help?