UDP socket leaks

[tubzby]

I have a program working as a proxy to convert RTSP stream to WebRTC stream so I can visit my cameras on the browser, normally a connection is closed when I close the tab, and I'm monitoring the ICE state to close PeerConnection, every time I call PeerConnection.close().await it complains:

Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  INFO avlib::pc_wrapper: close
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  WARN webrtc::peer_connection::peer_connection_internal: Failed to accept RTCP SessionSRTP has been closed
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  WARN webrtc::peer_connection::peer_connection_internal: Failed to accept RTP SessionSRTP has been closed
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate tcp4 host 192.168.0.28:9: the agent is closed
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 host 192.168.0.28:63422: the agent is closed
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 srflx 58.250.221.39:63422 related 192.168.0.28:63422: the agent is closed
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  WARN webrtc_ice::agent::agent_internal: [controlled]: Failed to close candidate udp4 relay 112.74.41.31:64537 related 58.250.221.39:63422: the agent is closed
Aug 28 14:46:56 debian10.toybrick rtsp2webrtc[23297]:  INFO webrtc_ice::agent::agent_internal: [controlled]: Setting new connection state: Closed

and all the UDP socket is never closed, I have confirmed that by lsof -n -p $pid.

[tubzby]

From the output of tokio-console, there are 2 tasks in warning state: "2 tasks have lost their waker", this might be the cause of leaks.

---EDIT 1-------

After adding some debug log, it seems there are an orphan task that never quit:

https://github.com/webrtc-rs/webrtc/blob/master/dtls/src/conn/mod.rs#L322

[tubzby]

I'm stuck, but with some findings:

1. turn/src/client/mod.rs: Client created in gather_candiates_relay was never dropped.
2. PeerConnection Agent AgentInternal were dropped.
3. Candidate.close() was called, but for UdpSocket, it does nothing, it has to be dropped.

It might related to the screenshot above, for some reason, the task created at agent_gather.rs:773 lost its waker, so the UdpSocket never released.

[tubzby]

Update:

I have been using Arc::strong_count to monitor Arc<CandidateBase> and find out that the count is 6.

I created a tokio task to spy on it, so the actual reference count is 5 which causing this problem.

[tubzby]

Finally, It leads to one possible line:

webrtc/ice/src/agent/agent_transport.rs

Line 244 in 23bbc1f

async fn close(&self) -> std::result::Result<(), util::Error> {

AgentConn::checklist should be cleared on close.

But there is still 1 leaking UdpSocket.

[rainliu]

Great catch, could you submit a PR to fix it? Thanks

[tubzby]

No problem, will submit a PR if there are no more leaks.

[tubzby]

Anyway, I was doing Arc::strong_count to tackle this, is there a better way to do that?

[tubzby]

Addition to: #608 (comment)

AgentConn.selected_pair should also be cleared.

[tubzby]

webrtc/ice/src/agent/mod.rs

Line 498 in 23bbc1f

//TODO: hook up _close_query_signal_tx to Agent or Candidate's Close signal?

mdns.query is never quit.

@rainliu There's a TODO comment line here, what's your suggestion?

[tubzby]

DTLSConn task stucked:

webrtc/dtls/src/conn/mod.rs

Line 322 in 23bbc1f

tokio::spawn(async move {

[tubzby]

I'm quite confused about why we should release these structures manually, is there a reference loop for Arc?

[dbidwell94]

I'm also seeing some odd behavior here. I have to manually impl Drop for my struct so the connections can be dropped. But once a connection has been facilitated, this code never returns and is stuck forever on waiting for the RTCPeerConnection to finish its call to close()

impl<'a> Drop for P2PConnection<'a> {
    fn drop(&mut self) {
        futures::executor::block_on(async move {
            let _ = self.data_channel.close().await;
            println!("Data Channel has been closed");
            let _ = self.connection.close().await;
            println!("Connection has been closed");
        });
    }
}

[dbidwell94]

This is the code in question that I believe is causing hangups in my project: https://github.com/dbidwell94/rust_p2p/blob/master/src/p2p_connection.rs#L187-L196

And the test in question is here: https://github.com/dbidwell94/rust_p2p/blob/master/src/p2p_connection.rs#L241

You should be able to easily pull the code down and run cargo test -- --no-capture to see the log outputs. Notice on the test in question, it drops the data_channel, but it never finishes dropping the RTCPeerConnection.

[mutexd]

@dbidwell94 I see that you're using futures::executor:block_on in the drop. The catch is that futures::executor::block_on would block the current caller until the future has completed. In the test, there is only one thread unless it's specified. You could use #[tokio::test(flavor = "multi_thread")] in your test and see if the problem persists.

[dbidwell94]

@dbidwell94 I see that you're using futures::executor:block_on in the drop. The catch is that futures::executor::block_on would block the current caller until the future has completed. In the test, there is only one thread unless it's specified. You could use #[tokio::test(flavor = "multi_thread")] in your test and see if the problem persists.

Confirmed, the tests now pass. Thanks for that!