IPv6 Rejoin Subflow

[Sakirnth-Z]

Hello

I have a dual-stack (IPv4+IPv6) setup with two interfaces.
When I do a mptcp connection it does establish a connection and creates a subflow, like expected.
But if I disconnect one interface and add it back as mptcp endpoint the IPv6 endpoint does not join as subflow again.
I have this problem only when I am connecting with IPv6.

I am testing this with a patched openssh version 9.7p1 to support mptcp. The same version is installed on both client and server.
https://github.com/openssh/openssh-portable/pull/335/files

When I establish and remove interface1 from the session I get this output with: ip mptcp monitor

[         CREATED] token=7b01f344 remid=0 locid=0 saddr6=$interface1 daddr6=$interface_server sport=38124 dport=22
[     ESTABLISHED] token=7b01f344 remid=0 locid=0 saddr6=$interface1 daddr6=$interface_server sport=38124 dport=22
[  SF_ESTABLISHED] token=7b01f344 remid=0 locid=4 saddr6=$interface2 daddr6=$interface_server sport=39705 dport=22 backup=0 ifindex=4
[       SF_CLOSED] token=7b01f344 remid=0 locid=0 saddr6=$interface1 daddr6=$interface_server sport=38124 dport=22 backup=0

But interface1 did not join the connection after it was back on and the endpoint created in subflow mode.

Here are some additional informations:

limit on the client:

add_addr_accepted 0 subflows 8

limit on the server:

add_addr_accepted 4 subflows 4

I am using kernel version 6.6.15 and following mptcp settings (same on server and client):

/proc/sys/net/mptcp/add_addr_timeout
120
/proc/sys/net/mptcp/allow_join_initial_addr_port
1
/proc/sys/net/mptcp/checksum_enabled
0
/proc/sys/net/mptcp/enabled
1
/proc/sys/net/mptcp/pm_type
0
/proc/sys/net/mptcp/scheduler
default
/proc/sys/net/mptcp/stale_loss_cnt
4

Thanks and greetings
Saki

[matttbe]

Hi Saki,

Sorry for the delay, it looks like we missed the issue.

But if I disconnect one interface and add it back as mptcp endpoint the IPv6 endpoint does not join as subflow again.

I didn't try to reproduce it yet, but it looks strange it works in v4 and not in v6, there should not be any difference.
How are the MPTCP endpoints added in your case? Is it possible they are added while the IPv6 addresses are not ready to be used yet? (e.g. DAD is still in progress).

Also, just to be sure, do you have the same issue if you increase the 'subflows' limit to 8 on the server side as well?

[Sakirnth-Z]

Hello

I didn't try to reproduce it yet, but it looks strange it works in v4 and not in v6, there should not be any difference. How are the MPTCP endpoints added in your case? Is it possible they are added while the IPv6 addresses are not ready to be used yet? (e.g. DAD is still in progress).

The default behaviour on my laptop is that Network Manager (Version 1.42.4 ) creates the endpoints.

connection.mptcp-flags: 0x0 (default) -> default /proc/sys/net/mptcp/enabled [1]

I disabled the NetworkManager MPTCP and did add it manually but the outcome was the same. And also I waited several minutes but the interface didn't join to rule out DAD.

Also, just to be sure, do you have the same issue if you increase the 'subflows' limit to 8 on the server side as well?

I did that but it had no effect.
limits on the server: add_addr_accepted 2 subflows 8

[1] https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-settings-nmcli.html

Thank you and kind regards
Saki

[matttbe]

The default behaviour on my laptop is that Network Manager (Version 1.42.4 ) creates the endpoints.

Nice, I didn't know it was enabled by default.

connection.mptcp-flags: 0x0 (default) -> default /proc/sys/net/mptcp/enabled [1]

Ah good, so it adds endpoints with the 'subflow' flag by default, that's good.

I disabled the NetworkManager MPTCP and did add it manually but the outcome was the same. And also I waited several minutes but the interface didn't join to rule out DAD.

Thank you for having tested!

By chance, did you try by setting a different MPTCP endpoint ID? Maybe the ID is still marked as used by mistake. But I don't see differences between v4 and v6 handling regarding that part.

And just to be sure, if you restart the connection after having reproduced the problem (disconnect/reconnect → attached endpoints are removed/re-added → attached v6 subflows are not re-established), the expected subflows are correctly created in v4 and v6, right?

One last thing, if you play with the interface of the second interface (not the one used by the initial subflow), do you have the same issue?

[Sakirnth-Z]

Sorry for the delay.

By chance, did you try by setting a different MPTCP endpoint ID? Maybe the ID is still marked as used by mistake. But I don't see differences between v4 and v6 handling regarding that part.

Network Manager creates the endpoint with different ID. I can test that manually without NetworkManager if you want.

And just to be sure, if you restart the connection after having reproduced the problem (disconnect/reconnect → attached endpoints are removed/re-added → attached v6 subflows are not re-established), the expected subflows are correctly created in v4 and v6, right?

Yes when I restart the connection all expected subflows are correctly created in both v4 and v6.

One last thing, if you play with the interface of the second interface (not the one used by the initial subflow), do you have the same issue?

Yes I have tested that too.

[matttbe]

Sorry for the delay.

No problem!

By chance, did you try by setting a different MPTCP endpoint ID? Maybe the ID is still marked as used by mistake. But I don't see differences between v4 and v6 handling regarding that part.

Network Manager creates the endpoint with different ID. I can test that manually without NetworkManager if you want.

Do you mean that when NM re-creates endpoints, the ID is the different from before?

(There are some corner cases that are being fixed, where the ID cannot be re-used, e.g. if there were no subflow linked to it, but that's not the case here)

And just to be sure, if you restart the connection after having reproduced the problem (disconnect/reconnect → attached endpoints are removed/re-added → attached v6 subflows are not re-established), the expected subflows are correctly created in v4 and v6, right?

Yes when I restart the connection all expected subflows are correctly created in both v4 and v6.

Thank you for having tested.

One last thing, if you play with the interface of the second interface (not the one used by the initial subflow), do you have the same issue?

Yes I have tested that too.

(I wanted to say "if you play with the interface of the second subflow")

And do you have the same issue?

In your case here, is the IPv6 subflow special somehow? e.g. is it the initial subflow? (do you start your connection in v4 or v6?)

Or do you have the full output of ip mptcp monitor showing all subflows of the same connection?

[Sakirnth-Z]

Do you mean that when NM re-creates endpoints, the ID is the different from before?

Yes I have copied my test case now. I unpluged wlp0s20f3 two times. The NM gives the same subflow ID before the unplug from the IPv6 address to the IPv4 address after the unplug. And after the reconnect the IPv6 gets a new subflow ID.

This was the test output of "ip mptcp endpoint show":

original state:
${IPV4_ADDR1} id 1 subflow dev enx
${IPV4_ADDR2} id 2 subflow dev wlp0s20f3
${IPV6_ADDR1} id 3 subflow dev enx
${IPV6_ADDR2} id 4 subflow dev wlp0s20f3

first unplug and reconnect:
${IPV4_ADDR1} id 1 subflow dev enx
${IPV6_ADDR1} id 3 subflow dev enx
${IPV4_ADDR2} id 4 subflow dev wlp0s20f3
${IPV6_ADDR2} id 5 subflow dev wlp0s20f3

second unplug and reconnect:
${IPV4_ADDR1} id 1 subflow dev enx
${IPV6_ADDR1} id 3 subflow dev enx
${IPV4_ADDR2} id 5 subflow dev wlp0s20f3
${IPV6_ADDR2} id 6 subflow dev wlp0s20f3

(There are some corner cases that are being fixed, where the ID cannot be re-used, e.g. if there were no subflow linked to it, but that's not the case here)

Ack.

Thank you for having tested.

You are welcome.

(I wanted to say "if you play with the interface of the second subflow")

And do you have the same issue?

Yes, no matter if I disconnect the interface of the first or second subflow the subflow does not rejoin the connection in v6.

In your case here, is the IPv6 subflow special somehow? e.g. is it the initial subflow? (do you start your connection in v4 or v6?)

I start my connection in v6. IMHO there is nothing special about the connection. Maybe the NM does configure something which does not work in this setting.

Or do you have the full output of ip mptcp monitor showing all subflows of the same connection?

I attached the "ip mptcp monitor" output in the first message, do you need more information?

[matttbe]

And after the reconnect the IPv6 gets a new subflow ID.

I think I'm a bit lost. Just to be sure:

• you have 2 interfaces, each with one v4 and one v6 address
• if you disconnect/reconnect any of these interface, only the subflow in v4 is re-established, not the one in v6
• endpoints are managed with NM, and the endpoint in v6 has a new ID (the one in v4 takes the one from v6)
• if you manually delete and re-add the problematic v6 endpoint with ip mptcp, readding it with a different ID, but the same subflow flag and dev option, you still don't have a new subflow, right?

I'm not able to reproduce it :-/ (but without NM)

Also, if you establish your initial connection in v6, I'm not sure to understand how the client will establish the connection in v4, because you set the add_addr_accepted to 0, no? Or was it mixed up, and the limits are set to 8?

What's the endpoint config on the server side? Only 2 endpoints, one in v4 and one in v6?

Could you get the output of sudo ss -Manit before and after an unplug and same after the replug to see if the counters look OK please?

There are some fixes ongoing around this, do you think you could compile a kernel using the code from this tag, and reproduce the issue please?

[matttbe]

One last thing, if you have:

• server:
  • endpoint signal v4
  • endpoint signal v6
• client:
  • endpoint subflow v4 iface 1
  • endpoint subflow v6 iface 1
  • endpoint subflow v4 iface 2
  • endpoint subflow v6 iface 2

and the initial connection in v6, only one v4 subflow will be created, because an ADD_ADDR is received and, if the add_addr_accepted limit is >0, will cause the creation of only one subflow using the routing rules. Except if you added the fullmesh flags?

Also, if you remove and re-add endpoint subflow v4 iface 2, it will not be re-established because it is only created thanks to the ADD_ADDR and the PM on the client side doesn't keep a ref to all received ADD_ADDR, see #496.

See https://www.mptcp.dev/pm.html for more details about how the PM is currently working.

[Sakirnth-Z]

* you have 2 interfaces, each with one v4 and one v6 address
* if you disconnect/reconnect any of these interface, only the subflow in v4 is re-established, not the one in v6
* endpoints are managed with NM, and the endpoint in v6 has a new ID (the one in v4 takes the one from v6)

Yes that's correct.

* if you manually delete and re-add the problematic v6 endpoint with `ip mptcp`, readding it with a different ID, but the same `subflow` flag and `dev` option, you still don't have a new subflow, right?

This works now. I can add new subflow endpoints manually with 'ip mptcp add' and the subflow gets created with the same ID and different ID. Either I tested it wrong last time or running on a newer kernel fixed this problem (now version 6.10.6).

Also, if you establish your initial connection in v6, I'm not sure to understand how the client will establish the connection in v4, because you set the add_addr_accepted to 0, no? Or was it mixed up, and the limits are set to 8?

No, you are right add_addr_accepted limit was 0 on the client and 4 for the server. I have v4 addresses but in my use case they are not used, only v6 endpoints. My aim was not to create a fullmesh setup but only a setup from several subflow endpoints from client to one server inerface (v6). Now when I remove and then add an active endpoint the subflow gets removed and is getting created again, like expected.

What's the endpoint config on the server side? Only 2 endpoints, one in v4 and one in v6?
The server has has only one endpoint for v6 which is configured as "implicit".

Could you get the output of sudo ss -Manit before and after an unplug and same after the replug to see if the counters look OK please?

On the server with NM:

mptcp ESTAB 0 0 [$server_endpoint_implicit_v6]:22 [$client_endpoint_subflow_v6_iface1]:53700
subflows:1 subflows_max:8 add_addr_accepted_max:8 remote_key token:110dd36b write_seq:12516047380791390004 snd_una:12516047380791390004 rcv_nxt:2575354605711998275

mptcp ESTAB 0 0 [$server_endpoint_implicit_v6]:22 [$client_endpoint_subflow_v6_iface1]:53700
add_addr_accepted:255 subflows_max:8 add_addr_accepted_max:8 remote_key token:110dd36b write_seq:12516047380791390004 snd_una:12516047380791390004 rcv_nxt:2575354605711998275

---

On the server without NM:

mptcp ESTAB 0 0 [$server_endpoint_implicit_v6]:22 [$client_endpoint_subflow_v6_iface1]:53900
subflows:1 subflows_max:8 add_addr_accepted_max:8 remote_key token:43b2ed30 write_seq:12847470930842621032 snd_una:12847470930842621032 rcv_nxt:11896971405719679327

mptcp ESTAB 0 0 [$server_endpoint_implicit_v6]:22 [$client_endpoint_subflow_v6_iface1]:53900
add_addr_accepted:255 subflows_max:8 add_addr_accepted_max:8 remote_key token:43b2ed30 write_seq:12847470930842695592 snd_una:12847470930842695592 rcv_nxt:11896971405719743855

Seems ok for me.

There are some fixes ongoing around this, do you think you could compile a kernel using the code from this tag, and reproduce the issue please?

I did compile and run the patched kernel version. I removed and added endpoints manually and with Network Manager it does work manually and with Network Manager not. Therefore same outcome.

One last thing, if you have:

* server:
  
  * endpoint signal v4
  * endpoint signal v6

* client:
  
  * endpoint subflow v4 iface 1
  * endpoint subflow v6 iface 1
  * endpoint subflow v4 iface 2
  * endpoint subflow v6 iface 2

server:

• endpoint implicit v6

and the initial connection in v6, only one v4 subflow will be created, because an ADD_ADDR is received and, if the add_addr_accepted limit is >0, will cause the creation of only one subflow using the routing rules. Except if you added the fullmesh flags?

No v4 subflow is created because the server has only one implicit v6 endpoint. Only one v6 subflow is created from endpoint subflow v6 iface 2 to endpoint v6 implicit to server.

Should I open a bug in Network Manager what you think?

[matttbe]

* if you manually delete and re-add the problematic v6 endpoint with `ip mptcp`, readding it with a different ID, but the same `subflow` flag and `dev` option, you still don't have a new subflow, right?

This works now. I can add new subflow endpoints manually with 'ip mptcp add' and the subflow gets created with the same ID and different ID. Either I tested it wrong last time or running on a newer kernel fixed this problem (now version 6.10.6).

It is possible the newer kernel fixed this problem, because quite a few issues around the path-manager have been fixed recently.

Also, if you establish your initial connection in v6, I'm not sure to understand how the client will establish the connection in v4, because you set the add_addr_accepted to 0, no? Or was it mixed up, and the limits are set to 8?

No, you are right add_addr_accepted limit was 0 on the client and 4 for the server. I have v4 addresses but in my use case they are not used, only v6 endpoints. My aim was not to create a fullmesh setup but only a setup from several subflow endpoints from client to one server inerface (v6). Now when I remove and then add an active endpoint the subflow gets removed and is getting created again, like expected.

Good!

What's the endpoint config on the server side? Only 2 endpoints, one in v4 and one in v6?

The server has has only one endpoint for v6 which is configured as "implicit".

If it is "implicit", it has not been added by NM, the kernel is using it because the address is being used by MPTCP for some reason (initial path?)

There are some fixes ongoing around this, do you think you could compile a kernel using the code from this tag, and reproduce the issue please?

I did compile and run the patched kernel version. I removed and added endpoints manually and with Network Manager it does work manually and with Network Manager not. Therefore same outcome.

Thank you for having checked.

One last thing, if you have:

* server:
  
  * endpoint signal v4
  * endpoint signal v6

* client:
  
  * endpoint subflow v4 iface 1
  * endpoint subflow v6 iface 1
  * endpoint subflow v4 iface 2
  * endpoint subflow v6 iface 2

server:

* endpoint implicit v6

So from what I understand, on the server side, no endpoints have been configured by NM, right?

and the initial connection in v6, only one v4 subflow will be created, because an ADD_ADDR is received and, if the add_addr_accepted limit is >0, will cause the creation of only one subflow using the routing rules. Except if you added the fullmesh flags?

No v4 subflow is created because the server has only one implicit v6 endpoint. Only one v6 subflow is created from endpoint subflow v6 iface 2 to endpoint v6 implicit to server.

Should I open a bug in Network Manager what you think?

Sorry, it is a bit difficult for me to follow, but if I understood correctly, the remaining issue is that NM didn't configure any endpoints on the server side, right? If yes, it might be good to check the NM config, checking if nothing has been missed by accident.

Please note that by default, I think NM would configure 'subflow' endpoints. Did you change something for NM to configure 'signal' endpoints instead? Maybe MPTCP support has been disabled by accident?