Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose API for PublicAddresses #212

Merged
merged 33 commits into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
8502645
identify: Expose API to inject listen addresses
lexnv Aug 20, 2024
e452745
tests: Check identify public addresses are propagated
lexnv Aug 20, 2024
0e42027
listen-addr: Add listen address struct to a separate module
lexnv Aug 28, 2024
890df79
listen-addr/tests: Check listen addr interface functionality
lexnv Aug 28, 2024
59668c9
listen-addr: Use a lock guard for custom iteration over the addresses
lexnv Aug 28, 2024
56ed02a
litep2p: Store shared listenAddresses on Litep2p object
lexnv Aug 28, 2024
a4bbf06
identify: Use the new interface of public addresses
lexnv Aug 28, 2024
f2a1c51
listen-addr: Check listen address contains p2p protocol
lexnv Aug 28, 2024
67b617d
Polish up the API
lexnv Aug 28, 2024
38d8ca3
listen-addr: Register lsiten addresses with adding the local peerID
lexnv Aug 28, 2024
2a45422
Adjust testing to the new interface
lexnv Aug 28, 2024
fc0c700
Use ListenAddresses everywhere
lexnv Aug 28, 2024
7f07e9d
listen-addr: Add better documentation
lexnv Aug 28, 2024
794eba5
listen-addr: Add contains and remove partial methods
lexnv Aug 29, 2024
5e9930a
Merge remote-tracking branch 'origin/master' into lexnv/indentify-con…
lexnv Aug 29, 2024
e11a996
Rename to ExternalAddresses for clarity
lexnv Aug 30, 2024
09b6fee
Refactor and adjust testing
lexnv Sep 2, 2024
c11f468
Merge remote-tracking branch 'origin/master' into lexnv/indentify-con…
lexnv Sep 2, 2024
68c76d2
Apply fmt
lexnv Sep 2, 2024
6161233
Fix cargo doc and clippy
lexnv Sep 2, 2024
dacd665
pub-addr: Rename API methods
lexnv Sep 3, 2024
1194a29
Introduce ListenAddresses object
lexnv Sep 3, 2024
ba6e198
Use listenAddresses added interface
lexnv Sep 3, 2024
e7a7d80
identify: Use user-provided, listen and public addresses
lexnv Sep 3, 2024
9ad762b
manager: Remove transport_manager::register_listen_address
lexnv Sep 3, 2024
d0ebcd2
Adjust testing
lexnv Sep 3, 2024
41c1e23
Fix documentation
lexnv Sep 3, 2024
fd23d15
address: Introduce insertion error for better reporting
lexnv Sep 3, 2024
be8d663
Remove ListenAddresses
lexnv Sep 3, 2024
af8fe60
Adjust testing
lexnv Sep 3, 2024
39153f1
Adjust testing
lexnv Sep 3, 2024
6b7b7b1
identify: Remove public addr from list config
lexnv Sep 4, 2024
a633b68
Merge branch 'master' into lexnv/indentify-confirmed-addresses
lexnv Sep 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 136 additions & 10 deletions src/protocol/libp2p/identify.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,14 @@ use crate::{

use futures::{future::BoxFuture, stream::FuturesUnordered, Stream, StreamExt};
use multiaddr::Multiaddr;
use parking_lot::RwLock;
use prost::Message;
use tokio::sync::mpsc::{channel, Sender};
use tokio_stream::wrappers::ReceiverStream;

use std::{
collections::{HashMap, HashSet},
sync::Arc,
time::Duration,
};

Expand All @@ -62,6 +64,54 @@ mod identify_schema {
include!(concat!(env!("OUT_DIR"), "/identify.rs"));
}

/// Set of the public addresses of the local node.
///
/// These addresses are reported to the identify protocol.
#[derive(Debug, Clone)]
pub struct IdentifyPublicAddresses {
inner: Arc<RwLock<HashSet<Multiaddr>>>,
}

impl IdentifyPublicAddresses {
/// Create new [`IdentifyPublicAddresses`].
fn new() -> Self {
Self {
inner: Arc::new(RwLock::new(HashSet::new())),
}
}

/// Add public address.
pub fn add(&self, address: Multiaddr) {
if address.is_empty() {
return;
}

self.inner.write().insert(address);
}

/// Remove public address.
pub fn remove(&self, address: &Multiaddr) {
self.inner.write().remove(address);
}

/// Get public addresses.
pub fn get_addresses(&self) -> Vec<Multiaddr> {
self.inner.read().iter().cloned().collect()
}

/// Get public addresses in the identify raw format.
fn get_raw_addresses(&self) -> Vec<Vec<u8>> {
self.inner.read().iter().map(|address| address.to_vec()).collect()
}

/// Extend public addresses.
pub fn extend(&self, addresses: impl IntoIterator<Item = Multiaddr>) {
self.inner
.write()
.extend(addresses.into_iter().filter(|address| !address.is_empty()));
}
}

/// Identify configuration.
pub struct Config {
/// Protocol name.
Expand All @@ -80,7 +130,7 @@ pub struct Config {
pub(crate) protocols: Vec<ProtocolName>,

/// Public addresses.
pub(crate) public_addresses: Vec<Multiaddr>,
pub(crate) public_addresses: IdentifyPublicAddresses,

/// Protocol version.
pub(crate) protocol_version: String,
Expand All @@ -98,20 +148,27 @@ impl Config {
protocol_version: String,
user_agent: Option<String>,
public_addresses: Vec<Multiaddr>,
) -> (Self, Box<dyn Stream<Item = IdentifyEvent> + Send + Unpin>) {
) -> (
Self,
IdentifyPublicAddresses,
Box<dyn Stream<Item = IdentifyEvent> + Send + Unpin>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could return IdentifyHandle which would hold IdentifyPublicAddresses and would implement Stream<Item = IdentifyEvent>, following the design of other protocols.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that idea, will certainly implement it! Thanks for the review!

Have put this on pending shortly, I have a hunch we'll need to expose an ExternalAddrConfirmed (like libp2p) to suffice other usecases (#211). Will wait for Dmitry to confirm that these are needed for the bootnodes on DHT🙏

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to have not responded here. We discussed #211 with @lexnv, and my idea was to introduce a globally accessible list of public addresses in litep2p. If some protocol needs external addresses, it can fetch them. This includes Identify and Kademlia protocols now.

As a first step, we can manually notify litep2p about confirmed external addresses from the client code. As a second step, we can move the heuristic of confirming external addresses seen by three or more peers from polkadot-sdk to litep2p. I have updated #211 to be more clear about this.

I don't think ExternalAddrConfirmed event will help with providing up-to-date peer addresses for local content providers in Kademlia from within litep2p, but the option to get addresses from the global list will.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your plan sounds good. The easiest place to store the external addresses is probably TransportManager, partly since there already exists code for storing known addresses via TransportManagerHandle and since every protocol has a handle to TransportManager, it should require little architectural changes.

) {
let (tx_event, rx_event) = channel(DEFAULT_CHANNEL_SIZE);
let identify_public_addresses = IdentifyPublicAddresses::new();
identify_public_addresses.extend(public_addresses);

(
Self {
tx_event,
public: None,
public_addresses,
public_addresses: identify_public_addresses.clone(),
protocol_version,
user_agent,
codec: ProtocolCodec::UnsignedVarint(Some(IDENTIFY_PAYLOAD_SIZE)),
protocols: Vec::new(),
protocol: ProtocolName::from(PROTOCOL_NAME),
},
identify_public_addresses,
Box::new(ReceiverStream::new(rx_event)),
)
}
Expand Down Expand Up @@ -183,7 +240,7 @@ pub(crate) struct Identify {
user_agent: String,

/// Public addresses.
listen_addresses: HashSet<Multiaddr>,
listen_addresses: IdentifyPublicAddresses,

/// Protocols supported by the local node, filled by `Litep2p`.
protocols: Vec<String>,
Expand All @@ -205,11 +262,13 @@ impl Identify {
config: Config,
listen_addresses: Vec<Multiaddr>,
) -> Self {
config.public_addresses.extend(listen_addresses);

Self {
service,
tx: config.tx_event,
peers: HashMap::new(),
listen_addresses: config.public_addresses.into_iter().chain(listen_addresses).collect(),
listen_addresses: config.public_addresses,
public: config.public.expect("public key to be supplied"),
protocol_version: config.protocol_version,
user_agent: config.user_agent.unwrap_or(DEFAULT_AGENT.to_string()),
Expand Down Expand Up @@ -269,11 +328,7 @@ impl Identify {
protocol_version: Some(self.protocol_version.clone()),
agent_version: Some(self.user_agent.clone()),
public_key: Some(self.public.to_protobuf_encoding()),
listen_addrs: self
.listen_addresses
.iter()
.map(|address| address.to_vec())
.collect::<Vec<_>>(),
listen_addrs: self.listen_addresses.get_raw_addresses(),
observed_addr,
protocols: self.protocols.clone(),
};
Expand Down Expand Up @@ -413,3 +468,74 @@ impl Identify {
}
}
}

#[cfg(test)]
mod tests {
use super::*;
use crate::{config::ConfigBuilder, transport::tcp::config::Config as TcpConfig, Litep2p};
use multiaddr::Multiaddr;

fn create_litep2p() -> (
Litep2p,
IdentifyPublicAddresses,
Box<dyn Stream<Item = IdentifyEvent> + Send + Unpin>,
) {
let (identify_config, pub_addr, identify) = Config::new(
"1.0.0".to_string(),
Some("litep2p/1.0.0".to_string()),
vec![Multiaddr::empty()],
);

let config = ConfigBuilder::new()
.with_tcp(TcpConfig {
listen_addresses: vec!["/ip6/::1/tcp/0".parse().unwrap()],
..Default::default()
})
.with_libp2p_identify(identify_config)
.build();

(Litep2p::new(config).unwrap(), pub_addr, identify)
}

#[tokio::test]
async fn update_identify_addresses() {
// Create two instances of litep2p
let (mut litep2p1, addr1, mut event_stream1) = create_litep2p();
let (mut litep2p2, _addr2, mut event_stream2) = create_litep2p();

let multiaddr: Multiaddr = "/ip6/::9/tcp/111".parse().unwrap();
// Litep2p1 is now listening on the new address.
addr1.add(multiaddr.clone());

// Dial `litep2p1`
litep2p2
.dial_address(litep2p1.listen_addresses().next().unwrap().clone())
.await
.unwrap();

tokio::spawn(async move {
loop {
tokio::select! {
_ = litep2p1.next_event() => {}
_event = event_stream1.next() => {}
}
}
});

loop {
tokio::select! {
_ = litep2p2.next_event() => {}
event = event_stream2.next() => match event {
Some(IdentifyEvent::PeerIdentified {
listen_addresses,
..
}) => {
assert!(listen_addresses.iter().any(|address| address == &multiaddr));
break;
}
_ => {}
}
}
}
}
}
2 changes: 1 addition & 1 deletion tests/conformance/rust/identify.rs
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ fn initialize_litep2p() -> (
) {
let keypair = Keypair::generate();
let (ping_config, ping_event_stream) = PingConfig::default();
let (identify_config, identify_event_stream) =
let (identify_config, _, identify_event_stream) =
IdentifyConfig::new("proto v1".to_string(), None, Vec::new());

let litep2p = Litep2p::new(
Expand Down
6 changes: 3 additions & 3 deletions tests/protocol/identify.rs
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ async fn identify_supported(transport1: Transport, transport2: Transport) {
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.try_init();

let (identify_config1, mut identify_event_stream1) = Config::new(
let (identify_config1, _, mut identify_event_stream1) = Config::new(
"/proto/1".to_string(),
Some("agent v1".to_string()),
Vec::new(),
Expand All @@ -75,7 +75,7 @@ async fn identify_supported(transport1: Transport, transport2: Transport) {
.with_libp2p_identify(identify_config1);
let config1 = add_transport(config_builder1, transport1).build();

let (identify_config2, mut identify_event_stream2) = Config::new(
let (identify_config2, _, mut identify_event_stream2) = Config::new(
"/proto/2".to_string(),
Some("agent v2".to_string()),
Vec::new(),
Expand Down Expand Up @@ -193,7 +193,7 @@ async fn identify_not_supported(transport1: Transport, transport2: Transport) {
.with_libp2p_ping(ping_config);
let config1 = add_transport(config_builder1, transport1).build();

let (identify_config2, mut identify_event_stream2) =
let (identify_config2, _, mut identify_event_stream2) =
Config::new("litep2p".to_string(), None, Vec::new());
let config_builder2 = ConfigBuilder::new()
.with_keypair(Keypair::generate())
Expand Down