Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(postgres): ssl for postgres #2473

Open
wants to merge 44 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
daa8cf2
SSL for postgres
Apr 8, 2024
53e2135
Add entrypoint wrapper
Apr 8, 2024
bd3360d
Add in init so we can test ssl+init path
Apr 8, 2024
c3fd69b
Remove unused fields from options
Apr 8, 2024
445a575
Remove unused consts
Apr 8, 2024
8814e8c
Separate entrypoint from ssl
Apr 9, 2024
2aecc1b
Use external cert generation
Apr 11, 2024
eac665d
Make entrypoint not-optional
Apr 12, 2024
302472c
Add docstring
Apr 12, 2024
3099790
Spaces to tab in entrypoint
Apr 24, 2024
e067d0f
Add postgres ssl docs
Apr 24, 2024
2c7f621
Remove WithEntrypoint
Apr 24, 2024
b9cd59b
Update docs/modules/postgres.md
bearrito Apr 24, 2024
ef017c9
Update docs/modules/postgres.md
bearrito Apr 24, 2024
fd3d3e5
Update docs/modules/postgres.md
bearrito Apr 24, 2024
cd1b63f
Update modules/postgres/postgres_test.go
bearrito Apr 24, 2024
c7eebeb
Update modules/postgres/postgres_test.go
bearrito Apr 24, 2024
a36eafa
Embed resources + Use custom conf automatically
Apr 24, 2024
7ea323a
Update docs/modules/postgres.md
bearrito Apr 26, 2024
0557c9d
Update docs/modules/postgres.md
bearrito Apr 26, 2024
5d0fa1e
Update docs/modules/postgres.md
bearrito Apr 26, 2024
7805e1f
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
c80fa83
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
e73f9a5
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
1be6a31
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
4e6c6d7
Revert to use passed in conf
Apr 26, 2024
e28361a
Update doc for required conf
Apr 26, 2024
8502649
Merge branch 'main' into feature/postgres-ssl
bearrito Apr 26, 2024
5d8598f
Error checking in the customizer
Apr 26, 2024
dffe996
Few formatting fix
Apr 26, 2024
53d9a1a
Use non-nil error when err is nil
Apr 26, 2024
146f47d
Update modules/postgres/postgres_test.go
bearrito Dec 23, 2024
94bd7f4
Update modules/postgres/postgres_test.go
bearrito Dec 23, 2024
dfa322e
Update modules/postgres/postgres.go
bearrito Dec 23, 2024
18ab8b7
Update modules/postgres/postgres.go
bearrito Dec 23, 2024
ab0e3ff
Update modules/postgres/postgres_test.go
bearrito Dec 23, 2024
9e15ac5
Addresses review modulo cleanup
Dec 24, 2024
c1cda1e
Remove unused type
Dec 24, 2024
3dfb0cb
Merge main into feature/postgres-ssl
Dec 24, 2024
ff6ce3e
Use ContainerCleanup
Dec 24, 2024
6adf44c
Lint pass
Dec 24, 2024
6fb5455
Add t.Helper and Linting
Dec 24, 2024
81a8d18
Remove SSLSetting struct, use raw paths
Dec 25, 2024
e980a90
Use single command for chown key material
Dec 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs/modules/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,34 @@ An example of a `*.sh` script that creates a user and database is shown below:

In the case you have a custom config file for Postgres, it's possible to copy that file into the container before it's started, using the `WithConfigFile(cfgPath string)` function.

This function can be used `WithSSLSettings` but requires your configuration correctly sets the SSL properties. See the below section for more information.

!!!tip
For information on what is available to configure, see the [PostgreSQL docs](https://www.postgresql.org/docs/14/runtime-config.html) for the specific version of PostgreSQL that you are running.

#### SSL Configuration

bearrito marked this conversation as resolved.
Show resolved Hide resolved
- Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

If you would like to use SSL with the container you can use the `WithSSLSettings`. This function accepts a `SSLSettings` which has the required secret material, namely the ca-certificate, server certificate and key. The container will copy this material to `/tmp/testcontainers-go/postgres/ca_cert.pem`, `/tmp/testcontainers-go/postgres/server.cert` and `/tmp/testcontainers-go/postgres/server.key`

This function requires a custom postgres configuration file that enables SSL and correctly sets the paths on the key material.

bearrito marked this conversation as resolved.
Show resolved Hide resolved
If you use this function by itself or in conjuction with `WithConfigFile` your custom conf must set the require ssl fields. The configuration must correctly align the key material provided via `SSLSettings` with the server configuration, namely the paths. Your configuration will need to contain the following:

```
ssl = on
ssl_ca_file = '/tmp/testcontainers-go/postgres/ca_cert.pem'
ssl_cert_file = '/tmp/testcontainers-go/postgres/server.cert'
ssl_key_file = '/tmp/testcontainers-go/postgres/server.key'
```

bearrito marked this conversation as resolved.
Show resolved Hide resolved
This function assumes the postgres user in the container is `postgres`

There is no current support for mutual authentication.

The `SSLSettings` function will modify the container `entrypoint`. This is done so that key material copied over to the container is chowned by `postgres`. All other container arguments will be passed through to the original container entrypoint.

### Container Methods

#### ConnectionString
Expand Down
1 change: 1 addition & 0 deletions modules/postgres/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
github.com/docker/go-connections v0.5.0
github.com/jackc/pgx/v5 v5.5.4
github.com/lib/pq v1.10.9
github.com/mdelapenya/tlscert v0.1.0
github.com/stretchr/testify v1.9.0
github.com/testcontainers/testcontainers-go v0.34.0

Expand Down
2 changes: 2 additions & 0 deletions modules/postgres/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mdelapenya/tlscert v0.1.0 h1:YTpF579PYUX475eOL+6zyEO3ngLTOUWck78NBuJVXaM=
github.com/mdelapenya/tlscert v0.1.0/go.mod h1:wrbyM/DwbFCeCeqdPX/8c6hNOqQgbf0rUDErE1uD+64=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
Expand Down
41 changes: 41 additions & 0 deletions modules/postgres/postgres.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package postgres
import (
"context"
"database/sql"
_ "embed"
"errors"
"fmt"
"io"
Expand All @@ -19,6 +20,9 @@ const (
defaultSnapshotName = "migrated_template"
)

//go:embed resources/customEntrypoint.sh
var embeddedCustomEntrypoint string

// PostgresContainer represents the postgres container type used in the module
type PostgresContainer struct {
testcontainers.Container
Expand Down Expand Up @@ -205,6 +209,43 @@ func WithSnapshotName(name string) SnapshotOption {
}
}

// WithSSLSettings configures the Postgres server to run with the provided CA Chain
// This will not function if the corresponding postgres conf is not correctly configured.
// Namely the paths below must match what is set in the conf file
func WithSSLCert(caCertFile string, certFile string, keyFile string) testcontainers.CustomizeRequestOption {
const defaultPermission = 0o600

return func(req *testcontainers.GenericContainerRequest) error {
const entrypointPath = "/usr/local/bin/docker-entrypoint-ssl.bash"

req.Files = append(req.Files,
testcontainers.ContainerFile{
HostFilePath: caCertFile,
ContainerFilePath: "/tmp/testcontainers-go/postgres/ca_cert.pem",
FileMode: defaultPermission,
},
testcontainers.ContainerFile{
HostFilePath: certFile,
ContainerFilePath: "/tmp/testcontainers-go/postgres/server.cert",
FileMode: defaultPermission,
},
testcontainers.ContainerFile{
HostFilePath: keyFile,
ContainerFilePath: "/tmp/testcontainers-go/postgres/server.key",
FileMode: defaultPermission,
},
testcontainers.ContainerFile{
Reader: strings.NewReader(embeddedCustomEntrypoint),
ContainerFilePath: entrypointPath,
FileMode: defaultPermission,
},
)
req.Entrypoint = []string{"sh", entrypointPath}

return nil
}
}

// Snapshot takes a snapshot of the current state of the database as a template, which can then be restored using
// the Restore method. By default, the snapshot will be created under a database called migrated_template, you can
// customize the snapshot name with the options.
Expand Down
86 changes: 86 additions & 0 deletions modules/postgres/postgres_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ package postgres_test
import (
"context"
"database/sql"
"errors"
"fmt"
"os"
"path/filepath"
"testing"
"time"
Expand All @@ -12,6 +14,8 @@ import (
"github.com/jackc/pgx/v5"
_ "github.com/jackc/pgx/v5/stdlib"
_ "github.com/lib/pq"
"github.com/mdelapenya/tlscert"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/testcontainers/testcontainers-go"
Expand All @@ -25,6 +29,40 @@ const (
password = "password"
)

func createSSLCerts(t *testing.T) (*tlscert.Certificate, *tlscert.Certificate, error) {
t.Helper()
tmpDir := t.TempDir()
bearrito marked this conversation as resolved.
Show resolved Hide resolved
certsDir := tmpDir + "/certs"

require.NoError(t, os.MkdirAll(certsDir, 0o755))

t.Cleanup(func() {
require.NoError(t, os.RemoveAll(tmpDir))
})

caCert := tlscert.SelfSignedFromRequest(tlscert.Request{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Host: "localhost",
Name: "ca-cert",
ParentDir: certsDir,
})

if caCert == nil {
return caCert, nil, errors.New("unable to create CA Authority")
}

cert := tlscert.SelfSignedFromRequest(tlscert.Request{
Host: "localhost",
Name: "client-cert",
Parent: caCert,
ParentDir: certsDir,
})
if cert == nil {
return caCert, cert, errors.New("unable to create Server Certificates")
}

return caCert, cert, nil
}

func TestPostgres(t *testing.T) {
ctx := context.Background()

Expand Down Expand Up @@ -171,6 +209,54 @@ func TestWithConfigFile(t *testing.T) {
defer db.Close()
}

func TestWithSSL(t *testing.T) {
ctx := context.Background()

caCert, serverCerts, err := createSSLCerts(t)
require.NoError(t, err)

container, err := postgres.RunContainer(ctx,
postgres.WithConfigFile(filepath.Join("testdata", "postgres-ssl.conf")),
postgres.WithInitScripts(filepath.Join("testdata", "init-user-db.sh")),
postgres.WithDatabase(dbname),
postgres.WithUsername(user),
postgres.WithPassword(password),
testcontainers.WithWaitStrategy(wait.ForLog("database system is ready to accept connections").WithOccurrence(2).WithStartupTimeout(5*time.Second)),
postgres.WithSSLCert(caCert.CertPath, serverCerts.CertPath, serverCerts.KeyPath),
)

testcontainers.CleanupContainer(t, container)
require.NoError(t, err)

connStr, err := container.ConnectionString(ctx, "sslmode=require")
require.NoError(t, err)

db, err := sql.Open("postgres", connStr)
require.NoError(t, err)
assert.NotNil(t, db)
defer db.Close()

result, err := db.Exec("SELECT * FROM testdb;")
require.NoError(t, err)
assert.NotNil(t, result)
}

func TestSSLValidatesKeyMaterialPath(t *testing.T) {
ctx := context.Background()

_, err := postgres.RunContainer(ctx,
postgres.WithConfigFile(filepath.Join("testdata", "postgres-ssl.conf")),
postgres.WithInitScripts(filepath.Join("testdata", "init-user-db.sh")),
postgres.WithDatabase(dbname),
postgres.WithUsername(user),
postgres.WithPassword(password),
testcontainers.WithWaitStrategy(wait.ForLog("database system is ready to accept connections").WithOccurrence(2).WithStartupTimeout(5*time.Second)),
postgres.WithSSLCert("", "", ""),
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: missing ContainerCleanup call


require.Error(t, err, "Error should not have been nil. Container creation should have failed due to empty key material")
}

func TestWithInitScript(t *testing.T) {
ctx := context.Background()

Expand Down
25 changes: 25 additions & 0 deletions modules/postgres/resources/customEntrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -Eeo pipefail


pUID=$(id -u postgres)
pGID=$(id -g postgres)

if [ -z "$pUID" ]
then
bearrito marked this conversation as resolved.
Show resolved Hide resolved
echo "Unable to find postgres user id, required in order to chown key material"
exit 1
fi

if [ -z "$pGID" ]
then
echo "Unable to find postgres group id, required in order to chown key material"
exit 1
fi

chown "$pUID":"$pGID" \
/tmp/testcontainers-go/postgres/ca_cert.pem \
/tmp/testcontainers-go/postgres/server.cert \
/tmp/testcontainers-go/postgres/server.key

/usr/local/bin/docker-entrypoint.sh "$@"
80 changes: 80 additions & 0 deletions modules/postgres/testdata/postgres-ssl.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# -----------------------------
# PostgreSQL configuration file
# -----------------------------
#
# This file consists of lines of the form:
#
# name = value
#
# (The "=" is optional.) Whitespace may be used. Comments are introduced with
# "#" anywhere on a line. The complete list of parameter names and allowed
# values can be found in the PostgreSQL documentation.
#
# The commented-out settings shown in this file represent the default values.
# Re-commenting a setting is NOT sufficient to revert it to the default value;
# you need to reload the server.
#
# This file is read on server startup and when the server receives a SIGHUP
# signal. If you edit the file on a running system, you have to SIGHUP the
# server for the changes to take effect, run "pg_ctl reload", or execute
# "SELECT pg_reload_conf()". Some parameters, which are marked below,
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
# "postgres -c log_connections=on". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: ms = milliseconds
# kB = kilobytes s = seconds
# MB = megabytes min = minutes
# GB = gigabytes h = hours
# TB = terabytes d = days


#------------------------------------------------------------------------------
# FILE LOCATIONS
#------------------------------------------------------------------------------

# The default values of these variables are driven from the -D command-line
# option or PGDATA environment variable, represented here as ConfigDir.

#data_directory = 'ConfigDir' # use data in another directory
# (change requires restart)
#hba_file = 'ConfigDir/pg_hba.conf' # host-based authentication file
# (change requires restart)
#ident_file = 'ConfigDir/pg_ident.conf' # ident configuration file
# (change requires restart)

# If external_pid_file is not explicitly set, no extra PID file is written.
#external_pid_file = '' # write an extra PID file
# (change requires restart)


#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = '*'
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
#port = 5432 # (change requires restart)
#max_connections = 100 # (change requires restart)

# - SSL -

ssl = on
ssl_ca_file = '/tmp/testcontainers-go/postgres/ca_cert.pem'
ssl_cert_file = '/tmp/testcontainers-go/postgres/server.cert'
#ssl_crl_file = ''
ssl_key_file = '/tmp/testcontainers-go/postgres/server.key'
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_dh_params_file = ''
#ssl_passphrase_command = ''
#ssl_passphrase_command_supports_reload = off


Loading