feat(swarm): Make executor for connection tasks explicit #3097

umgefahren · 2022-11-08T16:57:58Z

Description

Previously, the executor for connection tasks silently defaulted to a futures::executor::ThreadPool. This causes issues such as #2230.

With this patch, we force the user to choose, which executor they want to run the connection tasks on which results in overall simpler API with less footguns.

Closes #3068.

Links to any relevant issues

Open Questions

Is the API satisfactory?
Should the implementation of the executor live in the core, swarm, or somewhere else?
What is the correct location for the Swarm aliases

Change checklist

I will add a changelog entry once it's clear how broad the changes are.

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

thomaseizinger

Thanks for tackling this!

Please see inline comments :)

swarm/src/connection/pool.rs

core/src/lib.rs

swarm/src/lib.rs

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren · 2022-11-09T09:06:51Z

I can implement these changes, but I have a question: If we go back to trait objects here, which would be fine by me, that means I will remove all the genetics from the APIs it that's what you want. I'm asking because I read your issue a bit differently.

umgefahren · 2022-11-09T12:42:25Z

l implemented the suggested changes. It results in a lot of warnings, but I will remove them if your happy with the api.

thomaseizinger · 2022-11-09T22:22:01Z

I'm asking because I read your issue a bit differently.

Sorry if that was worded poorly! I tried to list multiple options there because I wasn't sure how things would turn out.

Breaking type inference is definitely a no-no in regards to ergonomics so adding feature-gated functions seems like a better approach!

thomaseizinger

Thanks! Some inline comments :)

core/Cargo.toml

core/src/lib.rs

examples/chat.rs

protocols/dcutr/examples/dcutr.rs

swarm/src/connection/pool.rs

swarm/src/lib.rs

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren · 2022-11-10T13:06:12Z

While we are on it: In the interest of #2166 we should add support for wasm bindgen futures.

Co-authored-by: Thomas Eizinger <[email protected]>

…p2p into executor-aware-swarm

thomaseizinger · 2022-11-10T21:32:42Z

While we are on it: In the interest of #2166 we should add support for wasm bindgen futures.

Oh that would be amazing actually!

core/Cargo.toml

swarm/src/connection/pool.rs

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren · 2022-11-11T11:57:19Z

Let me know as soon as your fine with the changes made. I will then start the process of replacing all usages of the deprecated API.

thomaseizinger

API looks good to me apart from one breaking change :)

swarm/src/lib.rs

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren · 2022-11-13T22:45:06Z

Unless you want me to remove #[derive(Default, Debug, Clone, Copy, Hash, PartialEq, Eq, PartialOrd, Ord)] I think there is nothing left to do for me here.

mxinden

Great work. Happy for this footgun to soon be gone.

@umgefahren could you please also create a pull request removing the deprecated methods? Once this is released we can then merge that second pull request.

swarm/src/connection/pool.rs

swarm/src/executor.rs

swarm/src/lib.rs

mergify · 2022-11-14T12:14:46Z

This pull request has merge conflicts. Could you please resolve them @umgefahren? 🙏

Co-authored-by: Max Inden <[email protected]>

umgefahren · 2022-11-14T13:17:06Z

This pull request has merge conflicts. Could you please resolve them @umgefahren? 🙏

That's probably the happiest merge conflict I ever had to resolve. Finally QUIC!!

mxinden

Follow up on inline discussion. Otherwise looks good to me.

swarm/src/connection/pool.rs

mxinden · 2022-11-15T12:56:20Z

swarm/CHANGELOG.md

 - Feature-gate `NetworkBehaviour` macro behind `macros` feature flag. See [PR 3055].

+- Make executor in Swarm constructor explicit. See [PR 3097].


Thank you for the great changelog entry.

mxinden · 2022-11-15T14:26:24Z

Again, thanks @umgefahren for the help!

mxinden · 2022-11-15T14:27:12Z

(I am sure many future users not falling into this trap will be very happy (unknowingly).)

mxinden · 2022-11-18T14:47:09Z

swarm/src/executor.rs

+
+#[cfg(feature = "tokio")]
+#[derive(Default, Debug, Clone, Copy)]
+pub(crate) struct TokioExecutor;


Why not expose TokioExecutor and AsyncStdExecutor? Folks that want to specify configs, e.g. dial_concurrency_factor need to use the SwarmBuilder. Instead of implementing Executor on their custom type wrapping e.g. the async-std executor, they could just reuse our implementation. Thoughts @umgefahren and @thomaseizinger?

Yes, I think you are right. If Thomas agrees I can open a PR real quick changing that and adding some documentation.

Alternatively, we could add the same set of with_ APIs.

I am okay either way. (I thought we had two layers of with_ APIs)

Can we just get rid of SwarmBuilder instead maybe? Is there a reason why any of the config values could not be set on Swarm itself?

Trying to upgrade iroh to laster master and blocked on this :/

Could you elaborate?

I nedd the SwarmBuilder and have to reimplement the Executor trait by copy pasting the one in here, to use tokio.

Can we just get rid of SwarmBuilder instead maybe? Is there a reason why any of the config values could not be set on Swarm itself?

@umgefahren Would you mind:

sending a hotfix PR that makes these types public

attempting to remove SwarmBuilder altogether by moving the current "builder" functions directly onto Swarm

LMK if you are not keen for the latter, then I'll try myself :)

Yes, I can do that (although I'm done for today)

…warmBuilder (#3155) Addresses issues raised here: #3097 (comment)

@umgefahren

With #3097, subscribing to the floodsub topic in `examples/chat-tokio.rs` was removed. I assume this was unintentional (cc @umgefahren), so this PR adds it back.

Previously, the executor for connection tasks silently defaulted to a `futures::executor::ThreadPool`. This causes issues such as libp2p#2230. With this patch, we force the user to choose, which executor they want to run the connection tasks on which results in overall simpler API with less footguns. Closes libp2p#3068.

umgefahren added 2 commits November 8, 2022 17:46

Initial prototype commit

ce2cb5c

Merge branch 'master' into executor-aware-swarm

6cb302f

thomaseizinger reviewed Nov 8, 2022

View reviewed changes

Update swarm/src/connection/pool.rs

b69dc35

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren added 3 commits November 9, 2022 10:24

Moving tokio and async std executors

12e0a32

Implement other changes

ad2ad9b

Formatted

92da6a2

umgefahren requested a review from thomaseizinger November 9, 2022 12:42

thomaseizinger reviewed Nov 9, 2022

View reviewed changes

Update swarm/src/connection/pool.rs

e7253d1

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren and others added 7 commits November 10, 2022 14:15

Update swarm/src/lib.rs

3a24540

Co-authored-by: Thomas Eizinger <[email protected]>

Implemented changes

98dc7ce

Merge branch 'executor-aware-swarm' of github.com:umgefahren/rust-lib…

38b21d7

…p2p into executor-aware-swarm

Merge branch 'master' into executor-aware-swarm

804b7d6

Fix tests

8fd5e56

Modified CI

ee9354c

Corrected some clippy issues

5381a2c

thomaseizinger mentioned this pull request Nov 10, 2022

swarm: Deprecate futures::executor::ThreadPool usage for connection tasks #3107

Closed

thomaseizinger reviewed Nov 10, 2022

View reviewed changes

core/Cargo.toml Outdated Show resolved Hide resolved

swarm/src/connection/pool.rs Show resolved Hide resolved

umgefahren and others added 3 commits November 10, 2022 23:22

Update core/Cargo.toml

dc5f1b7

Co-authored-by: Thomas Eizinger <[email protected]>

Merge branch 'master' into executor-aware-swarm

b82db0b

Implemented suggested changes

5365389

thomaseizinger reviewed Nov 11, 2022

View reviewed changes

swarm/src/lib.rs Show resolved Hide resolved

umgefahren and others added 3 commits November 13, 2022 23:30

Apply suggestions from code review

69fa671

Co-authored-by: Thomas Eizinger <[email protected]>

Implement small changes

f27fdf8

Merge branch 'master' into executor-aware-swarm

742f555

mxinden reviewed Nov 14, 2022

View reviewed changes

umgefahren and others added 3 commits November 14, 2022 14:09

Apply suggestions from code review

f61686d

Co-authored-by: Max Inden <[email protected]>

Merge branch 'master' into executor-aware-swarm

d193da3

Addressed Max's concerns.

30c5020

This was referenced Nov 14, 2022

swarm: Make API for creating a new Swarm executor aware #3068

Closed

swarm: Remove functions deprecated in 0.41.0 #3119

Closed

thomaseizinger changed the title ~~swarm: Make Swarm executor aware~~ feat(swarm): Make executor for connection tasks explicit Nov 14, 2022

Merge branch 'master' into executor-aware-swarm

f4ddadd

mxinden reviewed Nov 15, 2022

View reviewed changes

umgefahren added 2 commits November 15, 2022 14:19

Removed unnecessary inlines

51e98fb

Merge branch 'master' into executor-aware-swarm

1edb41d

mxinden approved these changes Nov 15, 2022

View reviewed changes

mxinden added the send-it label Nov 15, 2022

mergify bot merged commit d5ea93d into libp2p:master Nov 15, 2022

dmitry-markin mentioned this pull request Nov 18, 2022

Upgrade tokio to 1.22.0 and replace async-std with tokio paritytech/substrate#12646

Merged

mxinden reviewed Nov 18, 2022

View reviewed changes

umgefahren mentioned this pull request Nov 22, 2022

fix(swarm): Make executors pub and add convenience constructors for SwarmBuilder #3155

Merged

4 tasks

mergify bot pushed a commit that referenced this pull request Nov 23, 2022

fix(swarm): Make executors pub and add convenience constructors for S…

76dba17

…warmBuilder (#3155) Addresses issues raised here: #3097 (comment)

umgefahren mentioned this pull request Nov 25, 2022

chore(swarm): Remove deprecated functions #3170

Merged

4 tasks

elenaf9 mentioned this pull request Jan 2, 2023

fix(examples): subscribe to topic in chat-tokio #3296

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(swarm): Make executor for connection tasks explicit #3097

feat(swarm): Make executor for connection tasks explicit #3097

umgefahren commented Nov 8, 2022 •

edited by thomaseizinger

Loading

thomaseizinger left a comment

umgefahren commented Nov 9, 2022

umgefahren commented Nov 9, 2022

thomaseizinger commented Nov 9, 2022

thomaseizinger left a comment

umgefahren commented Nov 10, 2022

thomaseizinger commented Nov 10, 2022

umgefahren commented Nov 11, 2022

thomaseizinger left a comment

umgefahren commented Nov 13, 2022

mxinden left a comment

mergify bot commented Nov 14, 2022

umgefahren commented Nov 14, 2022

mxinden left a comment

mxinden Nov 15, 2022

umgefahren Nov 15, 2022

mxinden commented Nov 15, 2022

mxinden commented Nov 15, 2022

mxinden Nov 18, 2022

umgefahren Nov 18, 2022

thomaseizinger Nov 18, 2022

thomaseizinger Nov 18, 2022

dignifiedquire Nov 19, 2022

umgefahren Nov 19, 2022

dignifiedquire Nov 19, 2022

thomaseizinger Nov 21, 2022

umgefahren Nov 21, 2022

umgefahren Nov 22, 2022

		- Feature-gate `NetworkBehaviour` macro behind `macros` feature flag. See [PR 3055].

		- Make executor in Swarm constructor explicit. See [PR 3097].

feat(swarm): Make executor for connection tasks explicit #3097

feat(swarm): Make executor for connection tasks explicit #3097

Conversation

umgefahren commented Nov 8, 2022 • edited by thomaseizinger Loading

Description

Links to any relevant issues

Open Questions

Change checklist

thomaseizinger left a comment

Choose a reason for hiding this comment

umgefahren commented Nov 9, 2022

umgefahren commented Nov 9, 2022

thomaseizinger commented Nov 9, 2022

thomaseizinger left a comment

Choose a reason for hiding this comment

umgefahren commented Nov 10, 2022

thomaseizinger commented Nov 10, 2022

umgefahren commented Nov 11, 2022

thomaseizinger left a comment

Choose a reason for hiding this comment

umgefahren commented Nov 13, 2022

mxinden left a comment

Choose a reason for hiding this comment

mergify bot commented Nov 14, 2022

umgefahren commented Nov 14, 2022

mxinden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden commented Nov 15, 2022

mxinden commented Nov 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

umgefahren commented Nov 8, 2022 •

edited by thomaseizinger

Loading