From d31a8dbe332c230324dfe2f816800ab136044dc1 Mon Sep 17 00:00:00 2001 From: David Drysdale Date: Wed, 7 Feb 2018 15:57:03 +0000 Subject: [PATCH 1/3] Rewrite README to be more up-to-date/accurate --- README.md | 118 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 101 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 08bb5b2add..7cdacb000b 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,113 @@ -This is the beginnings of a certificate transparency log -client written in Go, along with a log scanner tool. +# Certificate Transparency: Go Code -You'll need go v1.8 or higher to compile. +[![Build Status](https://travis-ci.org/google/certificate-transparency-go.svg?branch=master)](https://travis-ci.org/google/certificate-transparency-go) +[![Go Report Card](https://goreportcard.com/badge/github.com/google/certificate-transparency-go)](https://goreportcard.com/report/github.com/google/certificate-transparency-go) +[![GoDoc](https://godoc.org/github.com/google/certificate-transparency-go?status.svg)](https://godoc.org/github.com/google/certificate-transparency-go) -# Installation +This repository holds Go code related to +[Certificate Transparency](https://www.certificate-transparency.org/) (CT). The +repository requires Go version 1.9. -This go code must be imported into your go workspace before you can -use it, which can be done with: + - [Repository Structure](#repository-structure) + - [Working on the Code](#working-on-the-code) + - [Rebuilding Generated Code](#rebuilding-generated-code) + - [Updating Vendor Code](#updating-vendor-code) + - [Running Codebase Checks](#running-codebase-checks) + +## Repository Structure + +The main parts of the repository are: + + - Encoding libraries: + - `asn1/` and `x509/` are forks of the upstream Go `encoding/asn1` and + `crypto/x509` libraries. We maintain separate forks of these packages + because CT is intended to act as an observatory of certificates across the + ecosystem; as such, we need to be able to process somewhat-malformed + certificates that the stricter upstream code would (correctly) reject. + Our `x509` fork also includes code for working with the + [pre-certificates defined in RFC 6962](https://tools.ietf.org/html/rfc6962#section-3.1). + - `tls` holds a library for processing TLS-encoded data as described in + [RFC 5246](https://tools.ietf.org/html/rfc5246). + - `x509util` provides additional utilities for dealing with + `x509.Certificate`s. + - CT client libraries: + - The top-level `ct` package (in `.`) holds types and utilities for working + with CT data structures defined in + [RFC 6962](https://tools.ietf.org/html/rfc6962). + - `client/` and `jsonclient/` hold libraries that allow access to CT Logs + via entrypoints described in + [section 4 of RFC 6962](https://tools.ietf.org/html/rfc6962#section-4). + - `scanner/` holds a library for scanning the entire contents of an existing + CT Log. + - Command line tools: + - `./client/ctclient` allows interaction with a CT Log + - `./scanner/scanlog` allows an existing CT Log to be scanned for certificates + of interest; please be polite when running this tool against a Log. + - `./x509util/certcheck` allows display and verification of certificates + - `./x509util/crlcheck` allows display and verification of certificate + revocation lists (CRLs). + - CT Personality for [Trillian](https://github.com/google/trillian): + - `trillian/` holds code that allows a Certificate Transparency Log to be + run using a Trillian Log as its back-end. + + +## Working on the Code + +Developers who want to make changes to the codebase need some additional +dependencies and tools, described in the following sections. The +[Travis configuration](.travis.yml) for the codebase is also useful reference +for the required tools and scripts, as it may be more up-to-date than this +document. + +### Rebuilding Generated Code + +Some of the CT Go code is autogenerated from other files: + + - [Protocol buffer](https://developers.google.com/protocol-buffers/) message + definitions are converted to `.pb.go` implementations. + - A mock implementation of the Trillian gRPC API is created with + [GoMock](https://github.com/golang/mock). + +Re-generating mock or protobuffer files is only needed if you're changing +the original files; if you do, you'll need to install the prerequisites: + + - `mockgen` tool from https://github.com/golang/mock + - `protoc`, [Go support for protoc](https://github.com/golang/protobuf) (see + documentation linked from the + [protobuf site](https://github.com/google/protobuf)) + +and run the following: ```bash -go get github.com/google/certificate-transparency-go/client -go get github.com/google/certificate-transparency-go/scanner -# etc. +go generate -x ./... # hunts for //go:generate comments and runs them ``` -# Building the binaries +### Updating Vendor Code + +The codebase includes a couple of external projects under the `vendor/` +subdirectory, to ensure that builds use a fixed version (typically because the +upstream repository does not guarantee back-compatibility between the tip +`master` branch and the current stable release). See +[instructions in the Trillian repo](https://github.com/google/trillian#updating-vendor-code) +for how to update vendored subtrees. + -To compile the log scanner run: +### Running Codebase Checks + +The [`scripts/presubmit.sh`](scripts/presubmit.sh) script runs various tools +and tests over the codebase. ```bash -go get github.com/google/certificate-transparency-go/scanner/scanlog -``` +# Install gometalinter and all linters +go get -u github.com/alecthomas/gometalinter +gometalinter --install -# Contributing +# Run code generation, build, test and linters +./scripts/presubmit.sh -When sending pull requests, please ensure that everything's been run -through ```gofmt``` beforehand so we can keep everything nice and -tidy. +# Run build, test and linters but skip code generation +./scripts/presubmit.sh --no-generate + +# Or just run the linters alone: +gometalinter --config=gometalinter.json ./... +``` From 720e11ab7d7c141cd0cb56f49d5d65e1fc9a4283 Mon Sep 17 00:00:00 2001 From: David Drysdale Date: Wed, 7 Feb 2018 16:14:04 +0000 Subject: [PATCH 2/3] Add more on Trillian CT personality --- README.md | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7cdacb000b..9c5c7e427c 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,37 @@ The main parts of the repository are: revocation lists (CRLs). - CT Personality for [Trillian](https://github.com/google/trillian): - `trillian/` holds code that allows a Certificate Transparency Log to be - run using a Trillian Log as its back-end. + run using a Trillian Log as its back-end -- see + [below](#trillian-ct-personality). + + +## Trillian CT Personality + +The `trillian/` subdirectory holds code and scripts for running a CT Log based +on the [Trillian](https://github.com/google/trillian) general transparency Log. + +The main code for the CT personality is held in `trillian/ctfe`; this code +responds to HTTP requests on the +[CT API paths](https://tools.ietf.org/html/rfc6962#section-4) and translates +them to the equivalent gRPC API requests to the Trillian Log. + +This obviously relies on the gRPC API definitions at +`github.com/google/trillian`; the code also uses common libraries from the +Trillian project for: + - exposing monitoring and statistics via an `interface` and corresponding + Prometheus implementation (`github.com/google/trillian/monitoring/...`) + - dealing with cryptographic keys (`github.com/google/trillian/crypto/...`). + +The `trillian/integration/` directory holds scripts and tests for running the whole +system locally. In particular: + - `trillian/integration/ct_integration_test.sh` brings up local processes + running a Trillian Log server, signer and a CT personality, and exercises the + complete set of RFC 6962 API entrypoints. + - `trillian/integration/ct_hammer_test.sh` brings up a complete system and runs + a continuous randomized test of the CT entrypoints. + +These scripts require a local database instance to be configured as described +in the [Trillian instructions](https://github.com/google/trillian#mysql-setup). ## Working on the Code @@ -65,8 +95,8 @@ Some of the CT Go code is autogenerated from other files: - [Protocol buffer](https://developers.google.com/protocol-buffers/) message definitions are converted to `.pb.go` implementations. - - A mock implementation of the Trillian gRPC API is created with - [GoMock](https://github.com/golang/mock). + - A mock implementation of the Trillian gRPC API (in `trillian/mockclient`) is + created with [GoMock](https://github.com/golang/mock). Re-generating mock or protobuffer files is only needed if you're changing the original files; if you do, you'll need to install the prerequisites: From 5b0d3e0a715a8039b445ad09721eafb543c7b4b8 Mon Sep 17 00:00:00 2001 From: David Drysdale Date: Wed, 7 Feb 2018 16:15:25 +0000 Subject: [PATCH 3/3] Add ToC entry --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 9c5c7e427c..6b71eaa987 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ This repository holds Go code related to repository requires Go version 1.9. - [Repository Structure](#repository-structure) + - [Trillian CT Personality](#trillian-ct-personality) - [Working on the Code](#working-on-the-code) - [Rebuilding Generated Code](#rebuilding-generated-code) - [Updating Vendor Code](#updating-vendor-code)