IPv4 broken on GoogleFi LTE

[ignoramous]

afontenot says,

After installing the most recent update (via fdroid) I can't access anything over IPv4 - so most websites. IPv6 routing works. Just for example, if I try to ping google.com in Termux, I see "no route to host". When I try with ping6 it works.

I tried reverting to the previous version and the problem goes away. I'm using the DNS-only mode.

This issue is only when using cell data. WiFi is fine.

---

This has also been known to happen with Google Fi / United States.

A couple things come to mind:

1. IPv4 routes aren't being routed by Rethink even when there's IPv4 connectivity (either through IPv4 in IPv6 or DS-Lite or DNS64/NAT64 or Teredo, etc)?
2. DNS endpoint does not respond to A records?

[ignoramous]

T-Mobile (Google Fi's network operator) is IPv6-only, with IPv4 support via 464Xlat (which also kind of relies on DNS64; ref).

[afontenot]

I have run some requested debugging commands via adb shell. I'm providing the output of these commands as attachments.

Some additional notes from further testing:

• When using System DNS mode, basic usage like browsing mostly works. Specifically, it works in cases where the system DNS returns AAAA records from a working DNS64 service. So stuff like curl -6 v4.ipv6test.app (connecting to a domain that has no AAAA record by design) magically works and returns my IPv4 address, but ping v4.ipv6test.app does not work, printing connect: No route to host. So switching to System DNS is a partial workaround for this issue, but won't solve every problem.

• When I switch the IP version in the settings to IPv4, the problem is resolved (even with Cloudflare as DNS). In this mode, IPv6 connections become unroutable, but now any IPv4 connections appear to get translated correctly. Losing IPv6 connectivity has broken some apps for me in the past, so this workaround is undesirable. Note: after switching to IPv4, the text on the main screen's drop down still says "protos: IPv6". Possible UI bug?

• The most striking thing to me from the logs is that the tun device doesn't get an IPv4 address in "auto" mode on LTE, but it does get one in IPv4 mode or when on WiFi (my home WiFI is dual stack).

Logs (edited to remove private data):

rethink_lte_log.txt
rethink_wifi_log.txt
rethink_lte_log_ipv4.txt

[ignoramous]

Thanks for sharing the output of various ip commands. They reveal what I suspected (a broken 464Xlat due to rethink mucking with the routes).

We are blocked by implementing a proper ICMP gateway before we can address this issue. Regardless, it is a priority item for us.

---

Specifically, [System DNS] works in cases where the system DNS returns AAAA records from a working DNS64 service.

We do have a way to capture the DNS64 prefix from the underlying DNS server, but it isn't going to be in any of the v053x releases. So, expect DNS64 to work whenever v054 is out (or, at least test the IPv6-only mode on Google Fi and let us know if it works as expected after v054 is out).

Note: after switching to IPv4, the text on the main screen's drop down still says "protos: IPv6". Possible UI bug?

The protos label there show the underlying network's protocols, not that of Rethink's (I mean, Rethink's protos would be exactly as chosen with the "Choose IP Version" setting, if that makes sense?).

The most striking thing to me from the logs is that the tun device doesn't get an IPv4 address in "auto" mode on LTE,

Yep, we don't add IPv4 routes in Auto mode if the underlying network's IPv4 routes are not globally route-able. Somehow (even if it should not), v4-rmnetdata (as address 192.0.0.4) doesn't get past those filters (even if it looks like it should). We don't have a 464xlat-enabled ISP where we can test this scenario out to truly identify where really we fall short. The ISPs here have by and large moved to DS-Lite (with CGNAT). You perhaps, iff you're up for it, testing it out for us is one (the only?) way out (:

rethink-app/app/src/main/java/com/celzero/bravedns/service/ConnectionMonitor.kt

Lines 272 to 290 in adf8b89

	val linkedProperties = networkProperties.linkedProperties
	val network = networkProperties.network
	// do not add network if there is no internet/is VPN
	if (hasInternet(network) == false || isVPN(network) == true) {
	return
	}
	linkedProperties.linkAddresses.forEach {
	if (it.scope != RT_SCOPE_UNIVERSE) return@forEach
	val address = IPAddressString(it.address.hostAddress?.toString()) ?: return@forEach
	if (address.isIPv6) {
	trackedIpv6Networks.add(network)
	} else {
	trackedIpv4Networks.add(network)
	}
	}

[afontenot]

We are blocked by implementing a proper ICMP gateway before we can address this issue. Regardless, it is a priority item for us.

That's good to know. I suspect quite a few users will run into this issue; this seems likely to be a dupe for instance: #558

You perhaps, iff you're up for it, testing it out for us is one (the only?) way out (:

Sure, I can test stuff as necessary. I did manage to get a self-signed build working with Android Studio, so if necessary I can build and test proposed patches.

[ignoramous]

In v053k (F-Droid and Play Store flavours should show up by next week), we've now defaulted to IPv4-only mode. For us to fully support 464Xlat and yet not break HappyEyeballs, we would have to cleverly handle the ICMP protocol. This will likely happen in v054 (due Nov / Dec 2022).

[ignoramous]

We've attempted fixing this (using "connectivity probes"; code) in v054c. Can someone using GoogleFi confirm if the problem reported here still persists?

[afontenot]

@ignoramous Nope, that's a fail here. Network mode is "auto". Sites with IPv6 addresses work, but I can't connect to any IPv4 address. Sites that are IPv4 only (e.g. twitter.com) fail.

adb_debug_log_lte.txt

[ignoramous]

Sigh. Two theories:

419: v4-rmnet_data1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 192.0.0.4/32 brd 192.0.0.4 scope global v4-rmnet_data1
valid_lft forever preferred_lft forever

The MTU of 1472 is lower than the 1500 set no the tun device by Rethink. May be that's the issue? But it doesn't explain why explicitly selecting IPv4 (from Choose IP version) works but Auto doesn't.

Note: after switching to IPv4, the text on the main screen's drop down still says "protos: IPv6". Possible UI bug?

Second is, Android doesn't even inform Rethink about the presence of the v4 network because the said v4 network doesn't meet the NET_CAPABILITY_INTERNET criteria? I am trying to find an app / adb shell¹ command that'd show network capability of all interfaces; output from which should somewhat help.

Footnotes

1. adb shell connectivity? ↩

[afontenot]

But it doesn't explain why explicitly selecting IPv4 (from Choose IP version) works but Auto doesn't.

Just for clarification - this revision to the auto mode behavior is supposed to switch to IPv4 automatically when dual-stack results in a broken connection, not fully fix 464xlat?

But it doesn't explain why explicitly selecting IPv4 (from Choose IP version) works but Auto doesn't.

Oops. I probably should have tested that after the upgrade, but in fact it appears to be broken for me. When I have Internet via LTE, and try to enable Rethink with IPv4 set, it hangs on "waiting" forever. If I have WiFi, enable it, and then switch off the WiFi, Rethink doesn't notice that I've switched networks (?!), so it says "protected", but I can't access anything over either IPv4 or IPv6! I didn't notice this because I've been using the auto mode for a while I think, since I'm on WiFi 99% of the time.

Setting the mode explicitly to IPv6 gets me "protected" under LTE, but (unsurprisingly I think) this means I can't contact IPv4 addresses. Probably this is why your new "auto" mode is negotiating IPv6 rather than IPv4?

If it would help, I could run the debugging commands while in IPv4 mode later today. Although note that the only way I'll be able to do that is to start in WiFi mode and then switch. (In fact I may have done this accidentally in the past, this might affect the debug logs in important ways...)

[ignoramous]

Just for clarification - this revision to the auto mode behavior is supposed to switch to IPv4 automatically when dual-stack results in a broken connection, not fully fix 464xlat?

Hm. In Auto mode, if you've got IPv4 connectivity, then IPv4-only servers (ex: twitter.com) should be reachable, as 464Xlat exists. In other words, you should see "protos: IPv4, IPv6" in Rethink's START / STOP bottom sheet, if routes were properly setup for egress via the 464Xlat interface.

If I have WiFi, enable it, and then switch off the WiFi, Rethink doesn't notice that I've switched networks (?!), so it says "protected"

I think, this is separate bug related to caching DNS responses (have you enable Advanced DNS filtering and/or DNS Booster by any chance?). When network is unavailable, Rethink will reply to DNS queries from its cache (this is infact cause of battery drain as apps frantically try to send TCP / UDP packets but can't, and since DNS works, apps are left further confused why TCP / UDP wouldn't work. We are fine tuning this bit up for v055 to avoid this scenario).

I can't access anything over either IPv4 or IPv6!

This is worrying. What does "protos" say? Also, anything in the Network Log at all or nothing shows up?

When I have Internet via LTE, and try to enable Rethink with IPv4 set, it hangs on "waiting" forever.

What does "protos" say in this case? This scenario is even more worrying! Essentially nullifies all our theories. Does switching to System DNS help at all in these scenarios? May be DNS64 isn't working as expected. Or Rethink is breaking outgoing NAT64 (this is a possibility, but I hope not).

Setting the mode explicitly to IPv6 gets me "protected" under LTE, but (unsurprisingly I think) this means I can't contact IPv4 addresses. Probably this is why your new "auto" mode is negotiating IPv6 rather than IPv4?

Looks like it, as in Rethink isn't able to ascertain IPv4 connectivity at all and hence doesn't setup routes for it. But that doesn't make sense since the 464Xlat interface should respond to ICMPv4 echo probes Rethink sends out.

If it would help, I could run the debugging commands while in IPv4 mode later today.

Since you're technical enough, you can observe packet-capture (Configure -> Settings -> Packet Capture -> Output to Logcat and grep for rdnspcap) in IPv4-only mode to see why Internet doesn't work?

I've asked @hussainmohd-a to create a test app to send across to you, so that we can get to the bottom of it before your patience wears out (:

[afontenot]

Edit: I'm leaving the following in case it turns out to be useful for some reason, but the issue is most likely unrelated, as I detail in the next comment.

I can't access anything over either IPv4 or IPv6!

This is worrying. What does "protos" say? Also, anything in the Network Log at all or nothing shows up?

When I have Internet via LTE, and try to enable Rethink with IPv4 set, it hangs on "waiting" forever.

What does "protos" say in this case? This scenario is even more worrying!

Hmm. Two points, after some more experimenting.

1. This seems to be flaky. It hung on "waiting" after several attempts earlier today, but just a few minutes ago when I tried it again it switched to "protected" eventually. Maybe I just didn't wait long enough before? (Maybe that connection check has to time out before the status gets updated?) In any case it's much slower than a successful activation of Rethink (e.g. on WiFi).

2. When in "protected" mode while on LTE and using the IPv4 setting, "protos" is blank (an empty string). The status line literally reads (active 3 min. ago; protos: ; type: Unmetered). I don't have connectivity over either IPv4 or IPv6.

I don't think this totally nullifies our theories because this is a new issue. There was a previous version of Rethink (not sure if it was the last version before the latest update or not) where I could at least get IPv4 connectivity with the IPv4 mode setting.

Does switching to System DNS help at all in these scenarios?

Nope, no change.

Also, anything in the Network Log at all or nothing shows up?

Not totally sure what you mean by the network log. Note that I am not using the firewall mode, only DNS, for these tests, so if this log is related to that, that explains why I don't see it. I do see a DNS log entry for sites I request, and furthermore these are successful requests (even for domains that should not be cached!).

you can observe packet-capture (Configure -> Settings -> Packet Capture -> Output to Logcat and grep for rdnspcap) in IPv4-only mode

pcap.txt doesn't look terribly useful to me, but maybe it's more obvious what is happening to you. This shows a few failed attempts to connect (hanging on "waiting"), and eventually I try curl -4 icanhazip.com in Termux at the end.

Maybe I'll try downgrading Rethink to see if I can pinpoint the regression. It's possible the issue I'm facing here in IPv4 mode isn't part of the larger issue with 464xlat.

[afontenot]

Ugh. This may have been a wild goose chase, sorry. I ended up rebooting the phone after downgrading didn't fix it, and the issue with IPv4 over LTE went away. So what I see is:

• Auto mode: IPv6 works, no IPv4.
• IPv4 mode: IPv4 works, no IPv6.

I'm not sure what could have caused this. Maybe it's that I have another VPN app (Wireguard) that I occasionally use to connect to my home network. I don't know how this could have caused this problem, since only one VPN app is ever active at a time. I haven't succeeded in recreating the issue.

[ignoramous]

Auto mode: IPv6 works, no IPv4.

With Configure -> Network -> Perform connectivity checks turned ON (if Choose IP version is set to Auto), one user has confirmed that both IPv6 and IPv4 work on networks that do 464Xlat: #1543 (comment)

Without it, only IPv6 does (like reported above).

The issue without Rethink doing its own connectivity checks (and instead relying on Android's) is documented here (where Android only reports IPv6 network but does not report the "stacked" IPv4 rmnet-data which does 464Xlat): https://groups.google.com/g/discuss-webrtc/c/ujdAFB25X3c / https://bugs.chromium.org/p/webrtc/issues/detail?id=9925