add probe for TLSv1.3 ECH (Encrypted ClientHello, ESNI replacement)

[eighthave]

I'm working with @sftcd on implementing the TLSv1.3 ECH standard as widely as possible. One thing that would be quite helpful would be to have data on whether anyone is blocking ECH currently. OONI seems like the best platform for that. I've never implemented an OONI probe. I have some funding to pay either myself or someone more qualified to do this.

I believe there is already an ESNI probe so that would be quite closely related.

[bassosimone]

@eighthave that's super interesting! If I understand it correctly, your project/implementation is to add support for that in OpenSSL, am I correct? We are currently mostly using Go code. I know https://github.com/cloudflare/go is a fork of Go that also implements ECH. Could using this be a (maybe not so cheap but certainly quite interesting) way of running these tests with OONI?

[eighthave]

Yes, we have implemented it in openssl, and are following the boringssl implementation. Then we also implement it in clients like curl, lighttpd, nginx, etc. It looks like that fork implements ECH already: https://github.com/cloudflare/go/blob/b0875fbf17405ccd0e3a57ec9fa0614d5e9be13c/src/crypto/tls/ech_test.go @sftcd do you know about how current Cloudflare's Go implementation is?

[sftcd]

On 20/04/2021 17:01, Hans-Christoph Steiner wrote: @sftcd do you know about how current Cloudflare's Go implementation is?

CF are up to date with draft-10. But for ooni you don't need a real ECH just to attempt with a GREASE'd one, so you just need the code-point and about the right number of random octets. S.

[eighthave]

I guess the tricky part is getting access to the TLS transaction at the right place to insert that code-point and random octets

[sftcd]

On 20/04/2021 17:57, Hans-Christoph Steiner wrote: I guess the tricky part is getting access to the TLS transaction at the right place to insert that code-point and random octets

Or just use CF's go implementation? If some specially crafted something were needed I'd be happy to try help with that, but I'd hope that's not the case. S.

…

[eighthave]

I've never done any Go before, so I'm not a great candidate to code this. @bassosimone how can I help make this happen?

[eighthave]

This is still interesting to us. Anything we can do to help? We have lots of example code in C and C++ for example.

[d1vyank]

I am interested in taking this up. Given ECH’s ‘do not stick out’ requirements and attempts by censors to block prior iterations of the protocol (ESNI), I think it is important to test early on if the draft version of ECH is subject to any network interference.

I propose to add an experimental test to probe-cli or miniooni that attempts a TLS connection with an encrypted Client Hello. I’ve looked into the two approaches discussed above:

• Use Cloudflare’s ECH implementation: This approach will require compiling OONI with Cloudflare's Go fork as they have implemented ECH in the standard library, and Go does not permit swapping out parts of the standard library (discussion here). I imagine this will require a new make command that will fetch and compile cloudflare/go and then build OONI with the forked version of Go. We can also use compiler directives to exclude the ECH test when regular versions of Go are used. Once built with the fork, the test can simply use crypto/tls to attempt an ECH connection to a server that supports it.
• Attempt a GREASE’d ECH connection: Since ECH connections are supposed to look identical to non-ECH connections, the protocol allows for a mocked ECH extension with random values to be present in the Client Hello. The test can use a library like refraction-networking/uTLS to get access to the Client Hello during the TLS handshake and insert a mock ECH extension.

I am leaning towards the GREASE approach as it seems like a less intrusive change than using a Go fork. Even though GREASE is not a complete ECH connection, it will provide valuable early evidence of interference to TLS connections using the ECH extension.

Happy to hear your thoughts!

[sftcd]

On 29/07/2022 07:38, Divyank Katira wrote: I am leaning towards the GREASE approach as it seems like a less intrusive change than using a Go fork. Even though GREASE is not a complete ECH connection, it will provide valuable early evidence of interference to TLS connections using the ECH extension.

That makes sense. Happy to provide samples etc if that helps and/or if you're testing against defo.ie, I can look at logs if debugging's needed. One suggestion - when you GREASE you should expect to get a retry_config in return that has a (or some) ECHConfig(s) that the server uses. It'd be interesting to record those and see if they differ from locale to locale perhaps. That'd be most interesting as help in figuring out any interop problems and likely less so in terms of exploring censorship as you should only see the retry_config after the server's authenticated. Happy to discuss more as you get to doing it. Cheers, S.

[kelmenhorst]

Update: Since we have ECH check in the tree for a few months now, we want to start collecting the first baseline measurements. We will start by adding it to the experimental nettest suite, with single static input.

[eighthave]

Great! Let me know if you need anything from us (DEfO). For example, we can recommend some sites to monitor, based on your parameters.

[kelmenhorst]

@eighthave Thanks, that would be great, what sites would you recommend then?

[eighthave]

https://crypto.cloudflare.com/cdn-cgi/trace and tls-ech.dev would probably be a good place to start.

we maintain a list here, you could try any of those https://github.com/sftcd/openssl/blob/ECH-draft-13c/esnistuff/smoke_ech.sh#L10

[eighthave]

We just published a HOWTO for how to set up dev/test servers using our TLS Encrypted ClientHello (ECH) enabled forks of OpenSSL, nginx and curl running on Debian. It should be very quick to get started using a new domain: https://guardianproject.info/2023/11/10/quick-set-up-guide-for-encrypted-client-hello-ech/

We just bought some dirt cheap domains and set up test sites. I think OONI devs will find this useful going forward. We might be able to provide a pre-configured VPS for you to mess with too.

[eighthave]

Just checking in on this. I see some things have been merged, but I can't find any references to ECH in https://explorer.ooni.org. Is there a way to see data on ECH yet?

Also, we have developer time and budget to spend on supporting OONI with this. Please let us know how best to jump in here.

[eighthave]

IIRC, these tests are written in Go. golang's core crypto/tls merged client ECH support, should be in golang 1.23. https://go-review.googlesource.com/c/go/+/578575

[eighthave]

We want to push this forward. One very useful improvement would be to save the RetryRequest after sending ECH GREASE. A full ECH connection with DNS tracking is the ideal.

FYI @irl has joined us in working on DEfO, and pointed us to the OONI ECH spec and other things in this codebase.

If you're on Matrix, we can help you interactively with any ECH questions or problems: https://matrix.to/#/#ech-dev:matrix.org

Go 1.23 has partial support for ECH on the Client Side: golang/go#63369. It still cannot fetch the ECH keys and there is no server-side support, but they would like to add it to 1.24: golang/go#68500 The client takes bytes for the config list: https://pkg.go.dev/crypto/tls@master#Config.EncryptedClientHelloConfigList

[sftcd]

RetryRequest

Just to clarify: that should be "retry_configs" - in the ECH protocol, if a client sends GREASE'd ECH then a server that does support ECH ought reply including an ECH extension in it's EncryptedExtensions reply with the payload of that being a set of ECHConfig values that can be used with that server (i.e. the stuff a browser would have gotten from an HTTPS RR in the DNS). If the OONI probe could record those values that would be useful data. (I'm not sure what golang APIs might allow doing that, but they may exist.) Going further, if the OONI probe could then use those values in a real ECH attempt (i.e. non-GREASE) and record that result, then that'd be a fine 2nd step forward once the golang TLS library has support for the client-side of ECH, and without having to code up all the HTTPS RR DNS handling.

[hellais]

I wrote a simple golang PoC of doing encrypted clienthello using the implementation part of golang 1.23. You can find it here: https://github.com/hellais/ech.

The code should cleaned up a bit before we integrate it into OONI Probe, but should show the basic pieces of how to perform the DoH lookup, parse the ECH configuration and pass it to go/tls.