crypto/tls: support NSS-formatted key log file

[shanemhansen]

In order to aid in debugging protocols which utilize tls (like http2), wireshark and other tools support reading a "key log" file as described here: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format

crypto/tls.Config struct should support logging keys to a io.Writer.

[bradfitz]

/cc @agl

[joneskoo]

I wish for this to be added, it would be so helpful debugging especially HTTP2. It is a trivial code change as demonstrated by my proof of concept. Obviously needs refining to bring this into standard library, but you can see that it's hardly any code to implement this.

I think the biggest question is how to enable this in a Go idiomatic fashion, especially through net/http. NSS uses environment variable SSLKEYLOGFILE for the file where to dump secrets.

As I understand the CLIENT_RANDOM based format is sufficient to cover everything, and I don't know if pre-1.8.0 Wireshark would be relevant for HTTP2 anyway.

I only demonstrated server handshake but client should be identical.

[bradfitz]

@joneskoo, I'd love to see this too. Please send a change once the Go 1.8 dev cycle opens after Go 1.7 is out in early August.

[shanemhansen]

I've been playing around trying to get this to work locally. No luck yet.
There seem to be a couple supported formats.

Wet to API I believe AGL suggested a io.Writer in the tls.Context.

I'd love to play with a branch of you have one.
On Jul 10, 2016 6:03 PM, "Brad Fitzpatrick" [email protected]
wrote:

@joneskoo https://github.com/joneskoo, I'd love to see this too. Please
send a change once the Go 1.8 dev cycle opens after Go 1.7 is out in early
August.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#13057 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAWY0ZGU3dxkbuMP0GJHVbYhDYcgGXp3ks5qUYhBgaJpZM4GWEKV
.

[agl]

This change isn't suitable until we're in the 1.8 cycle, but we'll get something working for 1.8.

Would adding an io.Writer member to tls.Config work for everyone?

[joneskoo]

@agl io.Writer works for me. I just finished trying out what it might look like; feel free to use or discard this implementation. Some trivial changes were needed in net/http as it does a shallow copy of the TLS config. Wonder what else. https://gist.github.com/joneskoo/254cd57e20acc7035d2c06658297b203

This is what I think it would look like for client and server:

• https://github.com/joneskoo/http2-keylog/blob/master/client.go#L28
• https://github.com/joneskoo/http2-keylog/blob/master/server.go#L36

I realize we're still waiting for the 1.8 to open, but this being open source, you can't stop me 🙈! But on all seriousness, I'll hold until 1.8 is open.

[bradfitz]

@joneskoo, please submit a CLA and send your change via Gerrit (which won't let you upload until you've done a CLA). We can't look at your code on Github until we know you've submitted a CLA.

@agl, I think we might want something a bit more generic than an io.Writer, which means we're locking ourselves into a format.

[bradfitz]

Actually, an io.Writer sounds fine. If anybody wanted to do something fancier, they can supply an io.Writer which parses the NSS keylog format back out, which is trivial enough.

[joneskoo]

@bradfitz If you wish to check out the change on https://github.com/joneskoo/http2-keylog I have signed the CLA and this repository is explicitly intended for submission to Go standard library when 1.8 opens. I need to find a slot to rebase and clean up for the Gerrit submission.

[jboelter]

Need tls clone method #15771

[joneskoo]

@agl, Submitted https://go-review.googlesource.com/27434 crypto/tls: add KeyLogWriter for debugging

[gopherbot]

CL https://golang.org/cl/27434 mentions this issue.

[joneskoo]

Can someone please explain what's the timeline going forward from this? When can I expect a review to happen and after that's all clear, how does the code actually end up in 1.8 development tree? The contributing doc doesn't really explain how review and merge works from a contributor's perspective, it only talks about the tools.

I've addressed Brad's suggestions already.

[bradfitz]

LGTM. I'll just submit it for now. @agl can do another pass later.

[joneskoo]

This should stay open until @agl gives this a look, right?

I'm expecting review comments to be on things I don't really know about. My change was "KISS", minimal changes.

• TLS features, e.g. renegotiation; should the key be logged there as well? What's the proper place for the trigger?
• Tests; I just looked at some existing tests and made the code reach the key logger in a trivial way; is this sufficient for standard library?

[bradfitz]

Sure, we can reopen.

The other thing I thought of after it was submitted was concurrency against the configured io.Writer in the presence of multiple connections. We probably just want some package-level mutex to guard all writes.

[joneskoo]

Good point about concurrent use. There's already a private mutex for session ticket keys; that shouldn't be reused? https://golang.org/src/crypto/tls/common.go#L392

TODO

• Does renegotiation needs special consideration? Now only handshakes are logged
• Is test coverage sufficient? How to test better?
• Protect against concurrent connections with same config

[bradfitz]

There's already a private mutex for session ticket keys; that shouldn't be reused?

I think I'd make a new, explicit one. Can you send that change first?

[joneskoo]

@bradfitz https://go-review.googlesource.com/29016 adds the mutex you suggested - should I also rename the existing "mutex" to "sessionTicketMutex"?

[joneskoo]

I did a "find references" search for masterFromPreMasterSecret and both of the two usages (client and server handshake) have the keylog call. Renegotiation appears to be part of the same handshake.

Can someone confirm that those are indeed the only two places where we get a new master secret for a connection?

And finally my question whether the test coverage is in fact sufficient?

[gopherbot]

CL https://golang.org/cl/29016 mentions this issue.

[joneskoo]

Are there other channels to communicate the availability of this when 1.8 is released? I can of course highlight it then personally. Is the changelog updated in this ticket or later in the release process? I don't think an example of usage would be justified in net/http given how few examples there currently are. I have an example use here: https://github.com/joneskoo/http2-keylog/blob/master/h2keylog-client/client.go#L41-L50. It's not as simple as I'd like because #17051 I'm still waiting for feedback whether we're done with this issue (tests, renegotiation). If yes, we can close from my point.

[bradfitz]

It'll be in the release notes. Adding an Example_keyLogWriter would be nice, though. Want to send one? I don't care about it looking out of place. We need more examples.

[joneskoo]

@bradfitz Done https://go-review.googlesource.com/29050

However since example is also a test, it fails because the master secrets are obviously unique for every connection and the one in the Output is not valid for any other execution. I expect this is addressed in code review.

I guess it's better to put this in crypto/tls anyway because the feature is really for any TLS, not just HTTP although that's probably what it's most useful with.

[gopherbot]

CL https://golang.org/cl/29050 mentions this issue.

[joneskoo]

@bradfitz ping? I used httptest in the example now, I guess you're the expert?

There's two variants, PS2 and PS4. I wish it was a bit cleaner; https://github.com/joneskoo/http2-keylog/blob/master/h2keylog-client/client.go#L41-L51 captures it better in my opinion.

I'd prefer an example with a real-world net/http with http2 which the current PS4 doesn't really do, which I raised in the other issue. I would gladly welcome someone else giving it a try and taking over, continuing on my work. I don't know when I'll be able to work on this next, could be some time.

[bradfitz]

@joneskoo, waiting for you to reply to feedback.

[joneskoo]

I will try to find time this weekend to complete the example.

[rsc]

@bradfitz, can you clarify: what is remaining on this issue?

[bradfitz]

I think it's done. I think @joneskoo had questions about renegotiations, but I don't think we care. We can address that later if it comes up.