Setup scaffolding for dynamic global settings

[josecelano]

Parent issue: #657

Context

Configuration injected when the Index starts is the only way to configure/customise the application. See below an example configuration injected via a config file and env vars.

Given that:

• We want to add many new options. See Index administration #657.
• We wanted to move some config options from the static configuration to the dynamic one.

Many configuration values were originally added to the file configuration because it was easy. However, there are some issues with that approach:

• You need to restart the service to apply the new configuration.
• It can only be done by sysadmin

It would be good to allow the admins (platform admin, not sysadmin responsible for the infrastructure) to change some settings. In fact, the admin is responsible for many of the current settings we put in the config file.

Naming

• dynamic global settings = runtime settings = admin settings
• static global settings = starting settings

Proposal

We need to add a new module/context for dynamic settings.

1. Add the new settings to this new module.
2. Migrate some static settings to dynamic settings. We have to do it without breaking changes. For example, we can insert in the database the values from the static config and priorize static settings. The sysadmin must remove the setting from the config file/env vars to start using the one in the database.

I see only these types of values:

• Boolean
• Plain text (UTF-8)
• Markdown

Some can be grouped, but that can be implemented in a second iteration.

We can have a new table, torrust_global_settings. The implementation details should go on the Index repo.

Example configuration in JSON

{
  "metadata": {
    "app": "torrust-index",
    "purpose": "configuration",
    "schema_version": "2.0.0"
  },
  "logging": {
    "threshold": "info"
  },
  "website": {
    "name": "Torrust",
    "demo": {
      "warning": "⚠️ Please be aware: This demo resets all data weekly. Torrents not complying with our Usage Policies will be removed immediately without notice. We encourage the responsible use of this software in compliance with all legal requirements."
    },
    "terms": {
      "page": {
        "title": "Usage Policies and Content Restrictions",
        "content": "\n# Usage Policies and Content Restrictions\n\nOur software is designed to support the distribution of legal, authorized content only. Users may only upload or share files that fall under the following categories:\n\n- **Open-Source Licenses:** Content licensed under recognized open-source licenses, allowing for free distribution and modification.\n- **Creative Commons Licenses:** Content released under Creative Commons licenses that permit sharing and distribution.\n- **Public Domain:** Content that is free of copyright restrictions and available for public use.\n\n**Prohibited Content:** Any content that infringes copyright, is subject to copyright protection, or is illegal under applicable laws is strictly prohibited. This includes but is not limited to copyrighted movies, music, software, books, and any other media.\n\n**Enforcement:** We reserve the right to remove any content that does not comply with these policies without notice. We may also take additional steps, including reporting violations to the relevant authorities, if necessary.\n\n"
      },
      "upload": {
        "content_upload_agreement": "I confirm that the content I am uploading is authorized, and I have read and agree to the terms."
      }
    }
  },
  "tracker": {
    "api_url": "http://localhost:1212/",
    "listed": false,
    "private": false,
    "token": "***",
    "token_valid_seconds": 7257600,
    "url": "udp://localhost:6969"
  },
  "net": {
    "base_url": null,
    "bind_address": "0.0.0.0:3001",
    "tsl": null
  },
  "auth": {
    "user_claim_token_pepper": "***",
    "password_constraints": {
      "max_password_length": 64,
      "min_password_length": 6
    }
  },
  "database": {
    "connect_url": "sqlite://data.db?mode=rwc"
  },
  "mail": {
    "from": "[email protected]",
    "reply_to": "[email protected]",
    "smtp": {
      "port": 25,
      "server": "",
      "credentials": {
        "password": "***",
        "username": ""
      }
    }
  },
  "image_cache": {
    "capacity": 128000000,
    "entry_size_limit": 4000000,
    "max_request_timeout_ms": 1000,
    "user_quota_bytes": 64000000,
    "user_quota_period_seconds": 3600
  },
  "api": {
    "default_torrent_page_size": 10,
    "max_torrent_page_size": 30
  },
  "registration": {
    "email": {
      "required": false,
      "verification_required": false
    }
  },
  "tracker_statistics_importer": {
    "port": 3002,
    "torrent_info_update_interval": 3600
  },
  "unstable": null
}

[josecelano]

We discussed about this issue in a meeting yesterday. We agreed on making it in small steps.

This is a new refined implementation.

Phase 1

The simplest way to introduce dynamic settings for the new features.

1. Create a new table torrust_index_global_settings

• Two text fields: key, value. value can be a long text.
• It will have only two records:
  • With key deafult-global-dynamic-settings.
  • With key global-dynamic-settings.

2. The value will be a JSON with all the settings.

• We need to create a database migration to seed the database with default values in both keys default-global-dynamic-settings, global-dynamic-settings

3. Define the JSON schema for this new dynamic settings.
4. Add a new section in the admin panel to edit global-dynamic-settings record.

• When the admin PUTs a new version, the application has to validate the schema.

5. A new Global Dynamic Settings service has to be created

• This service gets the data from the DB (global-dynamic-settings record).
• In-memory data is reloaded every time the admin PUTs a new version.
• We can inject this service in other service where we need the new settings.

Phase 2

Migrate some static settings from the config.toml (or env vars) file to dynamic settings.

We need to change the current configuration service. The current service for static configuration:

• Will allow overriding some settings values with dynamic settings. This will allow us to move some settings from static to dynamic.
• Will priorize dynamic settings when they are present in the dynamic configuration.
• Will maintain a list of settings values that can not be override (with the Figment path, like mandatory fields). This will not be needed as long as the JSON schema for the dynamic settings does not include these fields.

We allow all the fields in the static config we want to make dynamic.

NOTE: This has to be done manually by admins. The code will change the JSON schema but the admins have to include new fields manually in their databases. We can only add DB migrations for the default values. AE migration script could be provided.

Phase 3

Introduce a breaking change.

In version 4.0.0 remove from static config all the settings that are supported also by dynamic settings.

Those fields will be ignored when reading the TOML config.

Ideally, we only want to keep settings that require an application restart to be changed:

• Database configuration
• Socket addresses

Or are very unlikely to change or require infrastructure changes:

• Mail settings

And in general settings that are not ready to ve changed at runtime.

TODO

• Make a list of the static settings we want to move to dynamic.

NOTES

• We decided to use the DB to not introduce a dependency with the file system for application state.
• All dynamic settings have a default value.
• Static configuration is cloned in many places so it would be harder to refactor it to merge the dynamic values with static values with Figment. The new service for dynamic settings should not be cloned.

cc @da2ce7

[da2ce7]

Cameron's Proposal

torrust/torrust-index#778

Key Points

• Split Configuration into Four Separate Files or Environment Variables:

  1. Static-Private Configuration (/etc/torrust/index/private.toml or environment variable): Contains settings only known and set by the system administrator (e.g., DB Connection String).

  2. Static-Public Configuration (/etc/torrust/index/public.toml or environment variable): Contains settings set by the system administrator but may be shared with the service administrator (e.g., public socket, initial logging level).

  3. General Override (/etc/torrust/index/override.toml or environment variable): Contains settings overridden by the system administrator (e.g., obligatory privacy policy).

  4. General Default (/etc/torrust/index/default.toml or environment variable): Contains default settings set by the system administrator (e.g., default logo, welcome message).

• In-Database Settings File:

  • Store dynamic settings as a binary record in the database in json_utf8 format.
  • Service administrators can modify these settings without restarting the service.
  • Introduce new API endpoints to manage these settings (e.g., /admin/current, /admin/merged, /admin/default, /admin/override).

Migration Plan

To avoid breaking changes and provide a smooth transition to the new configuration system, the following phased migration plan is proposed:

Phase 1: Introduction of New Configuration Structure

• Implement Support for Multiple Configuration Sources:

  • Update the application to support reading configurations from the new files (private.toml, public.toml, override.toml, default.toml) and corresponding environment variables.
  • Ensure that the application can merge configurations from existing index.toml, new configuration files, and environment variables.
  • Define a clear precedence for configuration sources to prevent conflicts.

• Configuration Precedence:

  1. Highest Priority: Environment variables (TORRUST_INDEX_CONFIG_OVERRIDE_ prefix).
  2. Next: New configuration files (private.toml, public.toml, override.toml, default.toml).
  3. Lowest Priority: Existing index.toml file.

• Maintain Backward Compatibility:

  • Retain full support for the existing index.toml file to ensure current installations continue to function without modification.
  • Log warnings when deprecated settings are used from index.toml to encourage migration.

Phase 2: Gradual Migration of Settings

• Deprecate Settings in index.toml:

  • Identify settings that will be moved to the new configuration structure or dynamic settings in the database.
  • Mark these settings as deprecated in the documentation and output deprecation warnings in application logs when they are loaded from index.toml.

• Administrator Guidance:

  • Provide comprehensive documentation and migration guides to assist system administrators in transferring configurations from index.toml to the new files or environment variables.
  • Offer examples and best practices for setting up the new configuration structure.

• Optional Migration Tools:

  • Develop scripts or utilities to help administrators convert index.toml configurations into the new format or appropriate environment variables.

Phase 3: Enable Dynamic Configuration Through Database

• Implement In-Database Settings Management:

  • Create the database record for dynamic settings (json_utf8 format) to store service administrator configurations.
  • Deploy the new API endpoints for managing these settings:
    • /admin/current (GET/PUT)
    • /admin/merged (GET)
    • /admin/default (GET)
    • /admin/override (GET)

• Prioritize Dynamic Settings:

  • Modify the configuration loading logic so that dynamic settings from the database take precedence over static configurations.
  • Ensure that updates to the dynamic settings are reflected in the application's behavior without requiring a restart.

Phase 4: Deprecation and Removal of index.toml

• Announce Deprecation Timeline:

  • Communicate to administrators that support for index.toml will be deprecated in a future major release (e.g., version 4.0.0).
  • Provide a clear timeline and roadmap for the deprecation process.

• Final Migration Steps:

  • Encourage administrators to complete the migration to the new configuration structure before the deprecation date.
  • Offer assistance and resources during this period to address any migration challenges.

• Remove Support for index.toml:

  • In the specified future release, remove the ability to load configurations from index.toml.
  • Clean up associated code and resources related to the old configuration system.

Phase 5: Ongoing Support and Improvements

• Monitor and Support:

  • Continue to support administrators post-migration by addressing any issues and providing updates.
  • Gather feedback to improve the configuration system and administration interfaces.

• Iterative Enhancements:

  • Explore potential enhancements such as web-based configuration editors or additional administrative tools.
  • Consider further optimizations to configuration loading and management based on user feedback.

Advantages

• Clear Separation of Concerns:

  • Distinct configuration sources based on privacy levels and administrative responsibilities.

• Runtime Configuration:

  • Allows service administrators to update certain settings at runtime without restarting the application.

• Administrative Control:

  • Differentiates between system administrators (infrastructure-level settings) and service administrators (application-level settings).

• Flexible Configuration Sources:

  • Supports both file-based configurations and environment variables, accommodating different deployment environments.

• Managed Transition:

  • Provides a phased migration plan to prevent breaking changes and ease the transition for existing installations.

Considerations

• Complexity During Migration:

  • Managing multiple configuration sources temporarily may increase complexity.
  • Clear communication and documentation are essential to minimize confusion.

• Need for Comprehensive Documentation:

  • Detailed guides and resources are necessary to support administrators through the migration process.

• Potential for Overlapping Settings:

  • Careful handling of configuration precedence is required to prevent conflicts during the migration phase.

@josecelano