Dynamic Index Settings (Cameron's Proposal)

[da2ce7]

Related to torrust/torrust-index-gui#664

Current Situation

Torrust Index loads a read-only configuration upon startup; this configuration is static over the lifetime of the program.

Details

The Torrust Index reads its settings in TOML format from two places:

1. A file: /etc/torrust/index/index.toml, or a path overridden by the TORRUST_INDEX_CONFIG_TOML_PATH environment variable.
2. From the environment variable: TORRUST_INDEX_CONFIG_TOML.

If both configurations are present, the configuration supplied by the environment variable takes precedence.

The TOML format is overridden by configuration options supplied by the TORRUST_INDEX_CONFIG_OVERRIDE_ environment variable prefix.

This merged configuration is read-only for the lifetime of a torrust-index instance.

Configuration Types

There is a matrix of different types of configuration that the Torrust Index uses, for example:

	Instance	Runtime
Static	DB Connection String	API Socket
Dynamic	Logging Level	Index Policy

The current system only supports the Static-Instance configuration type, and all the runtime and dynamic configurations are treated as static-instance.

There is also a matrix of different administration levels for the configuration:

	Private	Public	General
System Admin	Only System Admin can set and view	Only the System Admin can set; others may see	Override value by default or Service Admin
Service Admin	Cannot access	Can view	Can set
Application Defaults	Not applicable	Default value	Default value

Proposal

Split the Configuration into Four Separate Files Controlled by the System Administrator

The same settings can be supplied via environment variables, similar to the current implementation. If both a configuration file and the corresponding environment variable are present, the environment variable takes precedence.

1. Static-Private Configuration

/etc/torrust/index/private.toml or TORRUST_INDEX_CONFIG_PRIVATE

This file will contain settings that are set by the system administrator and should only be known by the system administrator.

Example: the DB Connection String.

2. Static-Public Configuration

/etc/torrust/index/public.toml or TORRUST_INDEX_CONFIG_PUBLIC

This file will contain settings that are set by the system administrator and may be shared with the service administrator (or more).

Examples:

• Public socket of the service
• Initial logging level of the service

3. General Override

/etc/torrust/index/override.toml or TORRUST_INDEX_CONFIG_OVERRIDE

This file will contain settings that are overridden by the system administrator.

Examples:

• Obligatory Privacy Policy
• Runtime logging level of the service

4. General Default

/etc/torrust/index/default.toml or TORRUST_INDEX_CONFIG_DEFAULT

This file will contain default settings that are set by the system administrator.

Examples:

• Default logo
• Welcome message

Create an In-Database Settings File

As a record inside the database:

This record will contain a json_utf8 formatted entry with the settings supplied by the service administrator.

The following endpoints and actions will be created:

Endpoint	Action	Response
/admin/current	GET	200 OK: Returns the current configuration as supplied by the service administrator, or 204 No Content
/admin/current	PUT	200 OK: Sets new configuration; 400 Bad Request if the configuration is invalid
/admin/merged	GET	200 OK: Returns the current active configuration, including the public, overridden, current, and default settings, merged
/admin/default	GET	200 OK: Returns the default configuration, or 204 No Content
/admin/override	GET	200 OK: Returns any overrides for the configuration, or 204 No Content

[da2ce7]

Migration Plan

To ensure a smooth transition to the new configuration system without introducing breaking changes, the following migration plan is proposed:

Phase 1: Introduction of New Configuration Structure

• Support Multiple Configuration Sources:

  • Update the application to support reading configurations from the new files (private.toml, public.toml, override.toml, default.toml) and their corresponding environment variables:

    • TORRUST_INDEX_CONFIG_PRIVATE
    • TORRUST_INDEX_CONFIG_PUBLIC
    • TORRUST_INDEX_CONFIG_OVERRIDE
    • TORRUST_INDEX_CONFIG_DEFAULT

  • Ensure that the application can merge configurations from the existing index.toml, the new configuration files, and environment variables.

• Define Configuration Precedence:

  1. Highest Priority: Environment variables with the TORRUST_INDEX_CONFIG_OVERRIDE_ prefix (individual overrides).
  2. Next: Environment variables supplying the content of the new configuration files: private, public, override, and default, be if from a file, or supplied by an environmental variable.
  3. Lowest Priority: Existing index.toml file or the path specified by TORRUST_INDEX_CONFIG_TOML_PATH.

• Maintain Backward Compatibility:

  • Retain full support for the existing index.toml configuration file to ensure current installations continue to function without modification.

• Logging and Warnings:

  • Implement logging to inform administrators when configurations are loaded from deprecated sources (index.toml), encouraging migration to the new configuration structure.

Phase 2: Gradual Migration of Settings

• Deprecate Settings in index.toml:

  • Identify settings that will be moved to the new configuration structure or to dynamic settings in the database.

  • Mark these settings as deprecated in the documentation and output deprecation warnings in application logs when they are loaded from index.toml.

• Administrator Guidance:

  • Provide comprehensive documentation and migration guides to assist system administrators in transferring configurations from index.toml to the new files or environment variables.

  • Offer examples and best practices for setting up the new configuration structure.

• Optional Migration Tools:

  • Develop scripts or utilities to help administrators convert index.toml configurations into the new format or appropriate environment variables.

Phase 3: Enable Dynamic Configuration Through Database

• Implement In-Database Settings Management:

  • Create the database record for dynamic settings (in json_utf8 format) to store configurations supplied by the service administrator.

  • Deploy the new API endpoints to allow the service administrator to manage these settings at runtime.

• Prioritize Dynamic Settings:

  • Modify the configuration loading logic so that dynamic settings from the database take precedence over static configurations.

  • Ensure that updates to the dynamic settings are applied without requiring a service restart.

Phase 4: Deprecation and Removal of index.toml

• Announce Deprecation Timeline:

  • Communicate to administrators that support for index.toml will be deprecated in a future major release (e.g., version 4.0.0).

  • Provide a clear timeline and roadmap for the deprecation process.

• Final Migration Steps:

  • Encourage administrators to complete the migration to the new configuration structure before the deprecation date.

  • Offer assistance and resources during this period to address any migration challenges.

• Remove Support for index.toml:

  • In the specified future release, remove the ability to load configurations from index.toml.

  • Clean up associated code and resources related to the old configuration system.

Phase 5: Ongoing Support and Improvements

• Monitoring and Support:

  • Continue to support administrators post-migration by addressing any issues and providing updates.

  • Gather feedback to improve the configuration system and administrative interfaces.

• Iterative Enhancements:

  • Explore potential enhancements such as web-based configuration editors or additional administrative tools.

  • Consider further optimizations to configuration loading and management based on user feedback.