Add ja3 fields to Zeek SSL log

[andrewkroh]

What does this PR do?

Rename ja3s to tls.server.ja3s if present (requires zeek/salesforce/ja3 package package).
Rename ja3 to tls.client.ja3 if present (requires zeek/salesforce/ja3 package package).
Rename subject to tls.server.subject (instead of deleting it after parsing).
Rename not_valid_before to tls.server.not_before if present.
Rename not_valid_after to tls.server.not_after if present.

Fix dashboard reference for zeek.ssl.server.name that was using zeek.ssl.server_name.

Why is it important?

Improves the Zeek SSL dataset's ability to populate the TLS tab in the Security app.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Generate sample zeek.ssl logs with the zeek/salesforce/ja3 plugin installed and create test case.
• Add changelog.
• Update documentation to recommend the zeek/salesforce/ja3 plugin.
• Get sample scripts for adding the tls.server.not_after and tls.server.hash.sha1 to the zeek.ssl log and update the pipeline for those fields.

[softengchick]

This team uses the scripts from https://github.com/salesforce/ja3/tree/master/zeek for the ja3 fields. Here are the scripts for the other fields they add:

@load base/protocols/ssl
@load base/files/x509
@load base/utils/directions-and-hosts
module SSL;
export {
  redef record SSL::Info += {
    not_valid_before: time &optional &log;
    not_valid_after: time &optional &log;
  };
}
event ssl_established(c: connection) &priority=3
  {
  if ( ! c$ssl?$cert_chain || |c$ssl$cert_chain| == 0 ||
       ! c$ssl$cert_chain[0]?$x509 || ! c$ssl$cert_chain[0]?$sha1 )
    return;
  local cert = c$ssl$cert_chain[0]$x509$certificate;
  c$ssl$not_valid_before = cert$not_valid_before;
  c$ssl$not_valid_after = cert$not_valid_after;
  }

AND

module SSL;
export {
  redef record SSL::Info += {
    orig_certificate_sha1: string &optional &log;
    resp_certificate_sha1: string &optional &log;
  };
}
event file_hash(f: fa_file, kind: string, hash: string) {
  if (! ( kind == "sha1" && f?$source && f$source == "SSL" ))
    return;
	if ( |f$conns| != 1 )
		return;
	if ( ! f?$info || ! f$info?$mime_type )
		return;
	if ( ! ( f$info$mime_type == "application/x-x509-ca-cert" || f$info$mime_type == "application/x-x509-user-cert"
		 || f$info$mime_type == "application/pkix-cert" ) )
		return;
	local c: connection;
	for ( cid in f$conns )
	{
    c = f$conns[cid];
		if ( ! c?$ssl )
			return;
  	local chain: vector of string;
  	if ( f$is_orig )
  		chain = c$ssl$client_cert_chain_fuids;
  	else
  		chain = c$ssl$cert_chain_fuids;
  	if ( |chain| == 0 )
  		{
  		Reporter::warning(fmt("Certificate not in chain? (fuid %s)", f$id));
  		return;
  		}
  	# Check if this is the host certificate, if so, log hash by direction
  	if ( f$id == chain[0] ) {
      if ( f$is_orig )
        c$ssl$orig_certificate_sha1 = hash;
      else
        c$ssl$resp_certificate_sha1 = hash;
    }
	}
}

[softengchick]

This is a sample entry from their ssl.log:

{
    "ts": 1602179457.352156,
    "uid": "CK17Dl2SB8bZOVonSl",
    "id.orig_h": "10.7.9.101",
    "id.orig_p": 49228,
    "id.resp_h": "72.205.170.179",
    "id.resp_p": 443,
    "version": "TLSv12",
    "cipher": "TLS_RSA_WITH_AES_128_CBC_SHA256",
    "resumed": false,
    "established": true,
    "cert_chain_fuids": [
        "FOLwYQ6rs70bIMSf9"
    ],
    "client_cert_chain_fuids": [],
    "subject": "CN=afygrudeded.Alathal.owellle.bms,OU=Whe*swen and 4Ain ithe@lsath,O=Wofts Adererott FCP,L=Sanaa,C=YE",
    "issuer": "CN=afygrudeded.Alathal.owellle.bms,OU=Whe*swen and 4Ain ithe@lsath,O=Wofts Adererott FCP,L=Sanaa,C=YE",
    "validation_status": "self signed certificate",
    "ja3": "74927e242d6c3febf8cb9cab10a7f889",
    "ja3s": "80b3a14bccc8598a1f3bbe83e71f735f",
    "resp_certificate_sha1": "3dad8b55621b6b9c30679d9d61248dd132a83c94",
    "not_valid_before": 1562022424.0,
    "not_valid_after": 1577747224.0
}

*** They are happy to rename their fields output to whatever you see fit.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #21696 updated]

• Start Time: 2020-11-02T15:02:13.562+0000

• Duration: 66 min 27 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4475
Skipped	561
Total	5036

[andrewkroh]

I updated the pipeline and tests to include all the fields discussed. Now I want to test out the scripts and add adjust the documentation.

[andrewkroh]

The source of the second script is: https://github.com/rocknsm/rock-scripts/blob/1abcb137c3c0cb7bc1d54248d738255d2d6eb4ba/protocols/ssl/ssl-add-cert-hash.zeek

[adriansr]

LGTM

[elasticmachine]

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	4475
Skipped	561
Total	5036