Merge pull request #67 from kjsanger/feature/improved-config

Add TOML config file and update docs

README.md

@@ -11,77 +11,114 @@ serve data directly from iRODS.
 Sqyrrl is a available as and `amd64` binary for Linux, macOS and Windows,
 or as a Docker image. Copy the file to the desired location and run it.
 
-## Limitations
-
-Sqyrrl is an early development version and has the following limitations:
-
-- Does not authenticate users to the HTTP endpoint; anyone can access the data it serves.
-- Only serves files that have public access in iRODS.
-
 ## Running Sqyrrl
 
 Sqyrrl authenticates to iRODS using the standard method for an iRODS client i.e.
-using the iRODS environment file. It respects the `IRODS_ENVIRONMENT_FILE` environment
-variable, and if that is not set, it will look for the file in the standard location
-`$HOME/.irods/irods_environment.json`. Alternatively, command line option `--irods-env`
-may be used to set the environment file location explicitly.
+using the iRODS environment file.
 
-Since Sqyrrl will serve any data that it can access, it's important to use an iRODS user
-with appropriate authorization. The chosen iRODS user should have access only to public
-(unrestricted) data.
+Since Sqyrrl may serve any data that it can access, it's important to use an iRODS user
+with appropriate authorisation. In addition to the limitations imposed by the iRODS account
+used directly by the server, the server itself may be configured use OpenID Connect for
+HTTP client authentication. In this case, the user must also be authenticated by the OIDC
+provider and Sqyrrl will only serve data that the user has access to. OIDC user identity is
+mapped to an iRODS user account by user name.
+
+If the server is started without OIDC enabled, it will serve only data that is explicitly
+set to be readable by the `public` iRODS user.
 
 To start the server, use the following command:
 
 ```sh
+Configure and start the server.
+
 Usage:
   sqyrrl start [flags]
 
 Flags:
       --cert-file string          Path to the SSL certificate file
+      --config string             Path to a TOML configuration file
+      --enable-oidc               Enable OpenID Connect authentication
   -h, --help                      help for start
       --host string               Address on which to listen, host part (default "localhost")
       --index-interval duration   Interval at which update the index (default 1m0s)
-      --irods-env string          Path to the iRODS environment file (default "/Users/kdj/.irods/irods_environment.json")
+      --irods-env string          Path to the iRODS environment file
       --key-file string           Path to the SSL private key file
       --port string               Port on which to listen (default "3333")
 
 Global Flags:
       --log-level string   Set the log level (trace, debug, info, warn, error) (default "info")
-
 ```
 
+For additional options, use the `--help` flag.
+
 To stop the server, send `SIGINT` or `SIGTERM` to the process. The server will wait for
 active connections to close before shutting down.
 
-For additional options, use the `--help` flag.
 
-## Authentication - WARNING: This feature is not yet fully implemented
+### Configuration
+
+The preferred way to configure Sqyrrl is to use a TOML configuration file. This file may be
+specified using the `--config` flag. This file may be used to provide all the necessary
+configuration options and is the only way to pass secrets (OIDC client secret and iRODS
+password) to the server.
+
+An example configuration file is provided in the repository. The following fields are recognised:
+
+```toml
+Host = "<hostname>"
+Port  = "<port>"
+IRODSEnvFilePath = "<path to iRODS environment file>"
+IRODSPassword = "<iRODS password>"
+CertFilePath = "<path to SSL certificate file>"
+KeyFilePath = "<path to SSL private key file>"
+EnableOIDC  = true # Boolean value
+OIDCClientID = "<OICD client ID>"
+OIDCClientSecret = "<OIDC client secret>"
+OIDCIssuerURL = "<OIDC issuer URL>"
+OIDCRedirectURL = "<OIDC redirect URL>"
+IndexInterval = "<Interval string>" # e.g. "1m0s", "30s"
+```
+
+If `EnableOIDC` is set to false, the OIDC fields are not required and will be ignored, if present.
 
-Sqyrrl supports OpenID Connect for authentication. To enable OpenID Connect, use the
-`--enable-oidc` flag. The following environment variables are then required:
+Command line options and environment variables may also be used to configure the server
+for all settings except secrets. The configuration file has highest precedence, followed
+by command line options, and finally environment variables.
+
+Sqyrrl respects the `IRODS_ENVIRONMENT_FILE` environment variable, and if that is not set, it will
+look for the file in the standard location `$HOME/.irods/irods_environment.json`. Alternatively,
+command line option `--irods-env` may be used to set the environment file location explicitly.
+
+If an iRODS authentication file (default `~/.irods/.irodsA`) is present, Sqyrrl will use it
+and the iRODS password field is not required and will be ignored, if present.
+
+For backwards compatibility, it's possible to set some OIDC configuration options using
+environment variables. The following environment variables are recognised:
 
 - `OIDC_CLIENT_ID` - the client ID for the OIDC provider
-- `OIDC_CLIENT_SECRET` - the client secret for the OIDC provider
 - `OIDC_ISSUER_URL` - the URL of the OIDC provider
+- `OIDC_REDIRECT_URL` - the URL to which the OIDC provider should redirect after authentication
+
+## Authentication
+
+Sqyrrl supports OpenID Connect for authentication. To enable OpenID Connect, use the
+`EnableOIDC` field in the configuration file (or the `--enable-oidc` command line flag).
 
 Sqyrrl will then redirect users to the OIDC provider for authentication. The user will be
 redirected back to Sqyrrl after authentication.
 
-**Currently this feature does nothing more than enable the Login / Logout buttons on the
-home page.**
-
 ## iRODS authentication
 
 Sqyrrl uses the standard iRODS environment file to authenticate to iRODS. If the user has been
 authenticated with `iinit` before starting Sqyrrl, the server will use the existing iRODS auth
 file created by `iinit`. If the user has not been authenticated, Sqyrrl will require the iRODS
-password to be supplied using the environment variable `IRODS_PASSWORD`. Sqyrrl will then create
-the iRODS auth file itself, without requiring `iinit` to be used.
+password to be supplied using the `IRODSPassword` field of the Sqyrrl configuration file. Sqyrrl
+will then create  the iRODS auth file itself, without requiring `iinit` to be used.
 
 ## Running in a container
 
-When running Sqyrrl in a Docker container, configuration files (iRODS environment file, any
-existing auth file, SSL certificates) should be mounted into the container.
+When running Sqyrrl in a Docker container, configuration files (Sqyrrl configuration file, iRODS
+environment file, any existing auth file, SSL certificates) should be mounted into the container.
 
 The docker-compose.yml file in the repository contains an example configuration for running
 Sqyrrl in a container.
@@ -94,13 +131,13 @@ add a metadata attribute `sqyrrl:index` with value `1`. Data objects may be  gro
 on the page, under a title, known as a "category". To specify a category for a data object,
 add a metadata attribute `sqyrrl:category` with the  value being the category name.
 
-The home page will be re-indexed at the interval specified by the `--index-interval` flag. The
-home page auto-refreshes every 30 seconds.
+The home page will be re-indexed at the interval specified by the `IndexInterval` field in the
+configuration file (or the `--index-interval` command line flag). The home page auto-refreshes
+every 30 seconds.
 
 N.B. As go-irodsclient does not support metadata queries across federated zones, this feature
 is limited to data objects in the same zone as the iRODS user.
 
 ## Dependencies
 
 Sqyrrl uses [go-irodsclient](https://github.com/cyverse/go-irodsclient) to connect to iRODS. 
-

cmd/sqyrrl.go

@@ -19,6 +19,7 @@ package cmd
 
 import (
 	"fmt"
+	"github.com/BurntSushi/toml"
 	"io"
 	"os"
 	"strings"
@@ -35,9 +36,10 @@ import (
 var mainLogger = zerolog.New(zerolog.ConsoleWriter{Out: os.Stderr})
 
 type cliFlags struct {
-	certFilePath string // Path to the certificate file
-	keyFilePath  string // Path to the key file
-	envFilePath  string // Path to the iRODS environment file
+	certFilePath   string // Path to the certificate file
+	keyFilePath    string // Path to the key file
+	envFilePath    string // Path to the iRODS environment file
+	configFilePath string // Path to a TOML configuration file
 
 	host  string // Address to listen on, host part
 	level string // Logging level
@@ -48,9 +50,7 @@ type cliFlags struct {
 	enableOIDC bool // Enable OpenID Connect authentication
 }
 
-var cliFlagsSelected = cliFlags{
-	host: "localhost",
-}
+var cliFlagsSelected = cliFlags{}
 
 // configureRootLogger configures the root logger for the application. It sets up common
 // fields for the application name, version, and process ID, and it sets the default log
@@ -118,18 +118,78 @@ func printHelp(cmd *cobra.Command, args []string) {
 	}
 }
 
-func startServer(cmd *cobra.Command, args []string) error {
+func startServer(cmd *cobra.Command, args []string) (err error) { // NRV
 	logger := configureRootLogger(&cliFlagsSelected)
 
-	return server.ConfigureAndStart(logger, server.Config{
-		Host:          cliFlagsSelected.host,
-		Port:          cliFlagsSelected.port,
-		CertFilePath:  cliFlagsSelected.certFilePath,
-		KeyFilePath:   cliFlagsSelected.keyFilePath,
-		EnvFilePath:   cliFlagsSelected.envFilePath,
-		EnableOIDC:    cliFlagsSelected.enableOIDC,
-		IndexInterval: cliFlagsSelected.indexInterval,
-	})
+	var config server.Config
+	if cliFlagsSelected.configFilePath != "" {
+		var tomlData []byte
+		if tomlData, err = os.ReadFile(cliFlagsSelected.configFilePath); err != nil {
+			return err
+		}
+
+		_, err = toml.Decode(string(tomlData), &config)
+		if err != nil {
+			return err
+		}
+		logger.Info().Str("path", cliFlagsSelected.configFilePath).
+			Str("config", fmt.Sprintf("%v", config)).Msg("Config loaded")
+		config.ConfigFilePath = cliFlagsSelected.configFilePath
+	}
+
+	if cliFlagsSelected.host != "" {
+		config.Host = cliFlagsSelected.host
+		logger.Info().Str("host", config.Host).Msg(
+			"Configured host overridden on command line")
+	}
+	if cliFlagsSelected.port != "" {
+		config.Port = cliFlagsSelected.port
+		logger.Info().Str("port", config.Port).Msg(
+			"Configured port overridden on command line")
+	}
+	if cliFlagsSelected.certFilePath != "" {
+		config.CertFilePath = cliFlagsSelected.certFilePath
+		logger.Info().Str("path", config.CertFilePath).Msg(
+			"Configured certificate file path overridden on command line")
+	}
+	if cliFlagsSelected.keyFilePath != "" {
+		config.KeyFilePath = cliFlagsSelected.keyFilePath
+		logger.Info().Str("path", config.KeyFilePath).Msg(
+			"Configured key file path overridden on command line")
+	}
+	if cliFlagsSelected.envFilePath != "" {
+		config.IRODSEnvFilePath = cliFlagsSelected.envFilePath
+		logger.Info().Str("path", config.IRODSEnvFilePath).Msg(
+			"Configured iRODS environment file path overridden on command line")
+	}
+	if cliFlagsSelected.enableOIDC {
+		config.EnableOIDC = cliFlagsSelected.enableOIDC
+		logger.Info().Bool("enabled", config.EnableOIDC).Msg(
+			"Configured OpenID Connect authentication overridden on command line")
+	}
+	if cliFlagsSelected.indexInterval != 0 {
+		config.IndexInterval = cliFlagsSelected.indexInterval
+		logger.Info().Dur("interval", config.IndexInterval).Msg(
+			"Configured index interval overridden on command line")
+	}
+
+	err = server.Configure(logger, &config)
+	if err != nil {
+		return err
+	}
+
+	var srv *server.SqyrrlServer
+	srv, err = server.NewSqyrrlServer(logger, &config)
+	if err != nil {
+		return err
+	}
+
+	err = srv.Start()
+	if err != nil {
+		return err
+	}
+
+	return err
 }
 
 func CLI() {
@@ -152,10 +212,10 @@ func CLI() {
 		RunE:  startServer,
 	}
 	startCmd.Flags().StringVar(&cliFlagsSelected.host,
-		"host", "localhost",
+		"host", "",
 		"Address on which to listen, host part")
 	startCmd.Flags().StringVar(&cliFlagsSelected.port,
-		"port", "3333",
+		"port", "",
 		"Port on which to listen")
 	startCmd.Flags().StringVar(&cliFlagsSelected.certFilePath,
 		"cert-file", "",
@@ -164,8 +224,11 @@ func CLI() {
 		"key-file", "",
 		"Path to the SSL private key file")
 	startCmd.Flags().StringVar(&cliFlagsSelected.envFilePath,
-		"irods-env", server.LookupIRODSEnvFilePath(),
+		"irods-env", "",
 		"Path to the iRODS environment file")
+	startCmd.Flags().StringVar(&cliFlagsSelected.configFilePath,
+		"config", "",
+		"Path to a TOML configuration file")
 	startCmd.Flags().DurationVar(&cliFlagsSelected.indexInterval,
 		"index-interval", server.DefaultIndexInterval,
 		"Interval at which update the index")

config/sqyrrl.toml

@@ -0,0 +1,11 @@
+Host = "0.0.0.0"
+Port  = "3333"
+IRODSEnvFilePath = "/app/config/app_irods_environment.json"
+CertFilePath = "/app/config/localhost.crt"
+KeyFilePath = "/app/config/localhost.key"
+EnableOIDC  = true
+OIDCClientID = "0oafha8j3cQCmfxRP417"
+OIDCClientSecret = "pjDn3100j2eJaqg9uOYYZ3uDkHdHHQI9ku4Nkn9vx_6twZELwA_euVQeIg3iyVoJ"
+OIDCIssuerURL = ""
+OIDCRedirectURL = ""
+IndexInterval = "60s"

docker-compose.yml

@@ -20,15 +20,9 @@ services:
         context: .
         dockerfile: Dockerfile
       command: ["start",
-                "--host", "0.0.0.0",
-                "--port", "3333",
-                "--cert-file", "/app/config/localhost.crt",
-                "--key-file", "/app/config/localhost.key",
-                "--irods-env", "/app/config/app_irods_environment.json",
-                "--enable-oidc",
-                "--index-interval", "60s",
+                "--config", "/app/config/sqyrrl.toml",
                 "--log-level", "trace"]
-      # Set the following environment variables in a .env file (files named .env
+      # The following environment variables may be set in a .env file (files named .env
       # are declared in .gitignore):
       #
       # If no iRODS auth file is provided:
@@ -38,9 +32,12 @@ services:
       # And if using OIDC:
       #
       # OIDC_CLIENT_ID
-      # OIDC_CLIENT_SECRET
       # OIDC_ISSUER_URL
       # OIDC_CALLBACK_URL
+      #
+      # The OIDC client secret may not be set in the environment. Instead, it should be
+      # provided in the TOML config file mounted into the container at the path specified
+      # by the --config option..
       env_file: .env
       ports:
         - "3333:3333"

go.mod

@@ -17,6 +17,7 @@ require (
 )
 
 require (
+	github.com/BurntSushi/toml v1.4.0 // indirect
 	github.com/aymerick/douceur v0.2.0 // indirect
 	github.com/go-jose/go-jose/v4 v4.0.2 // indirect
 	github.com/go-logr/logr v1.4.2 // indirect

go.sum

@@ -1,3 +1,5 @@
+github.com/BurntSushi/toml v1.4.0 h1:kuoIxZQy2WRRk1pttg9asf+WVv6tWQuBNVmK8+nqPr0=
+github.com/BurntSushi/toml v1.4.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
 github.com/alexedwards/scs/v2 v2.8.0 h1:h31yUYoycPuL0zt14c0gd+oqxfRwIj6SOjHdKRZxhEw=
 github.com/alexedwards/scs/v2 v2.8.0/go.mod h1:ToaROZxyKukJKT/xLcVQAChi5k6+Pn1Gvmdl7h3RRj8=
 github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=