Merge remaining internal changes

.github/workflows/deploy-pages.yml

@@ -11,6 +11,8 @@ jobs:
       name: github-pages
       url: ${{steps.deployment.outputs.page_url}}
     runs-on: ubuntu-latest
+    permissions:
+      id-token: write
     steps:
       - name: Checkout
         uses: actions/checkout@v4

README.md

@@ -3,14 +3,14 @@
 > 🚨🚨**Notice**🚨🚨
 > 
 > Configuration for the Application Study Tool has changed significantly in the v0.6.0 release. To
-update a legacy configuration, see [docs/config_migration.md](docs/config_migration.md).
+update a legacy configuration, see [pages/config_migration.md](pages/config_migration.md).
 
 ## Overview
 
 The Application Study Tool is intended to provide enhanced insights into (classic) BIG-IP products, leveraging best in class
 open source telemetry tools. The full installation includes:
 
-* Custom Instance of OpenTelemetry Collector with enhanced BIG-IP data receivers (data fetched via iControlRest) [Full List of Metrics Collected](docs/receiver_metrics.md).
+* Custom Instance of OpenTelemetry Collector with enhanced BIG-IP data receivers (data fetched via iControlRest) [Full List of Metrics Collected](pages/receiver_metrics.md).
 * Prometheus timeseries database for storing and querying collected data.
 * Grafana Instance with pre-configured dashboards for quick insights at the device and "fleet" levels.
 
@@ -54,7 +54,7 @@ docker-compose up
 ## Configuration
 
 For additional configuration management background, see
-[docs/config-management.md](docs/config-management.md).
+[pages/config-management.md](pages/config-management.md).
 The below assumes you're using the config_helper script for assisted management.
 
 
@@ -65,7 +65,7 @@ Application Study Tool config management relies on default configs in
 Settings in the bigip_receivers.yaml override those in ast_defaults.yaml.
 
 To update a legacy (pre v0.6.0) configuration, to the new scheme see
-[docs/config_migration.md](docs/config_migration.md)
+[pages/config_migration.md](pages/config_migration.md)
 
 ## Configure Default Device Settings
 
@@ -85,7 +85,7 @@ bigip_receiver_defaults:
   # The data_types that should be enabled or disabled.
   # DNS and GTM are disabled by default and users can enable those modules
   # on all devices by setting the below to true.
-  # A full list of data_types is in /docs/receiver_readme.md.
+  # A full list of data_types is in pages/receiver_readme.md.
   data_types:
     f5.dns:
       enabled: false
@@ -203,9 +203,9 @@ as SENSOR_ID and SENSOR_SECRET_TOKEN (see [.env-example](./.env-example) for exa
 
 ## Run The Configuration Helper
 The config helper script can be run natively or via docker to merge the default and device
-level configs into the final OTEL Collector config as follows:
+level configs into the final OTEL Collector config from the project root directory as follows:
 ```shell
-# Run the configuration generator
+# Run the configuration generator from the project root directory
 docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
 ```
 

config/README.md

@@ -4,5 +4,5 @@ Files in this directory can be used to configure aspects of the Application Stud
 
 For additional detail, see:
 
-* [Configuration Management](/docs/config_management.md) (new users start here)
-* [Pre v0.6.0 Config Migration](/docs/config_migration.md)
+* [Configuration Management](pages/config_management.md) (new users start here)
+* [Pre v0.6.0 Config Migration](pages/config_migration.md)

docs/config_management.md → pages/config_management.md

@@ -1,4 +1,4 @@
-# AST Configuration Management Detail
+# AST Configuration Management Overview
 
 ## Config Management Options
 In the post v0.6.0 management scheme, users can choose from one of the below options to manage
@@ -15,7 +15,7 @@ and [/services/otel_collector/receivers.yaml](/services/otel_collector/receivers
 3. Manual maintenance of the Otel Collector Config files in
 [/services/otel_collector/defaults](/services/otel_collector/defaults)
 
-### Using config_helper.py (Recommended For Most Users)
+## Using config_helper.py (Recommended For Most Users)
 With the included python script in [/src/config_helper.py](/src/config_helper.py), AST collector
 configuration is managed through 2 primary files:
 
@@ -144,11 +144,21 @@ bigip/1:
   username: SOME_OVERRIDE_ACCOUNT_NAME
 ```
 
-When the OTEL container is run, the default configs in
-[/services/otel_collector/defaults/](/services/otel_collector/defaults/) merge these files into
-the final configuration the OTEL Collector needs to run correctly.
+#### Run The Configuration Helper
+The config helper script can be run natively or via docker from the project root directory
+to merge the default and device level configs into the final OTEL Collector config as follows:
+```shell
+# Run the configuration generator from the project root directory
+docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
+```
+
+This will write 2 new files in the services/otel_collector directory:
+
+* `receivers.yaml` - The final list of scraper configs and their settings.
+* `pipelines.yaml` - The final pipeline configs that map receievers to output destinations
+(prometheus).
 
-### Manual Maintenance Of Receiver and Pipeline Files
+## Manual Maintenance Of Receiver and Pipeline Files
 The files mentioned above can be managed directly by users if they want to skip the config_helper
 script in favor of their own automation / templating. In this case, you just need to update the files:
 
@@ -182,7 +192,7 @@ Any of these files can be modified directly to update settings as desired (e.g.
 logging levels).
 
 
-### Manual Maintenance Of The OTEL Collector Config
+## Manual Maintenance Of The OTEL Collector Config
 You can also forgo all of the above config structure in favor of your own management scheme. If you're
 running with the base docker-compose file, you may need to modify the commands section to point at your
 own config files:

docs/config_migration.md → pages/config_migration.md

@@ -11,7 +11,7 @@ where users were unable to view and modify the files to tune parameters for thei
 In the new process, the raw otel configs are exposed in the /services/otel_collector directory where
 they can be managed manually, or through continued use of a refactored config_helper script.
 
-For additional detail on configuration management options in the post v0.6.0 scheme, please see [/docs/config_management.md](/docs/config_management.md)
+For additional detail on configuration management options in the post v0.6.0 scheme, please see [Config Management](./config_management.md)
 
 ## Migrating From pre v0.6.0 Configs
 There's a python script in /src/config_helper.py which will convert the original big-ips.json schema

pages/index.md

@@ -4,7 +4,7 @@
 The Application Study Tool is intended to provide enhanced insights into (classic) BIG-IP products, leveraging best in class
 open source telemetry tools. The full installation includes:
 
-* Custom Instance of OpenTelemetry Collector with enhanced BIG-IP data receivers (data fetched via iControlRest) [Full List of Metrics Collected](/docs/receiver_metrics.md).
+* Custom Instance of OpenTelemetry Collector with enhanced BIG-IP data receivers (data fetched via iControlRest) [Receiver Metrics Info](./receiver_metrics.md).
 * Prometheus timeseries database for storing and querying collected data.
 * Grafana Instance with pre-configured dashboards for quick insights at the device and "fleet" levels.
 
@@ -17,4 +17,9 @@ the Openetlemetry Collector can be configured to send data to existing productio
 
 ### Table of Contents
 
-- [Quick Start](./quickstart.md)
+- [Quick Start](./quickstart.md)
+- [Config Management Overview](./config_management.md)
+- [Config Migration (From v0.5.0 and Earlier)](./config_migration.md)
+- [Receiver Metrics Info](./receiver_metrics.md)
+- [Receiver Readme](./receiver_readme.md)
+- [Metric Obfuscation](./metric_obfuscation.md)

pages/quickstart copy.md

@@ -0,0 +1,210 @@
+# Quick Start
+
+## Getting Started
+
+### Prerequisites
+
+docker (or compatible) - [Installation Instructions](https://docs.docker.com/engine/install/)
+
+### Installation
+
+Clone the repo or download source tarball from the [release](https://github.com/f5devcentral/application-study-tool/releases) section.
+
+```shell
+# Clone the repo
+git clone https://github.com/f5devcentral/application-study-tool.git
+cd application-study-tool
+# Edit the following file with Grafana variables as required
+cp .env-example .env
+# Edit the following file with device secrets as required (see "Configure Device Secrets" below)
+cp .env.device-secrets-example .env.device-secrets
+# Edit the default settings for your environment as required
+# (see "Configure Default Device Settings" below)
+vi ./config/ast_defaults.yaml
+# Edit the config file with device / connection info
+# (see "Configure Devices To Scrape" below)
+vi ./config/big_receivers.yaml
+# Run the configuration generator
+docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
+# Start the tool
+docker-compose up
+```
+
+## Configure Default Device Settings
+
+Edit config/ast_defaults.yaml to reflect common values for your BIG-IPs:
+```yaml
+# These configs are applied to each entry in the bigip_receivers file
+# where they don't contain an equivalent / overriding entry.
+bigip_receiver_defaults:
+  # The time to wait between metric collection runs
+  collection_interval: 60s
+  # The username to login to the device with
+  username: admin
+  # The password (not recommended) or a reference to an env variable (recommended)
+  # Below tells the collector to look for an environment variable named
+  # BIGIP_PASSWORD_1
+  password: "${env:BIGIP_PASSWORD_1}"
+  # The data_types that should be enabled or disabled.
+  # DNS and GTM are disabled by default and users can enable those modules
+  # on all devices by setting the below to true.
+  # A full list of data_types is in /docs/receiver_readme.md.
+  data_types:
+    f5.dns:
+      enabled: false
+    f5.gtm:
+      enabled: false
+  # The TLS settings to use. Either a CA file must be specified or
+  # insecure_skip_verify set to true (not recommended).
+  tls:
+    # Secure TLS communication requires mounting the certificate bundle
+    # used to sign the BigIP certificates. Though not recommended, in the
+    # case of self-signed certificates or for testing purposes, you can skip
+    # this check by setting this field to true.
+    insecure_skip_verify: false
+    # The path to a CA File used to validate BIG-IP certificates. This is required
+    # if tls_insecure_skip_verify is set to false. See below for details.
+    ca_file: ""
+```
+
+## Configure Devices To Scrape
+Edit the device list in config/bigip_receivers.yaml:
+```yaml
+#### Values not explicitly configured here inherit values in ast_defaults.yaml.
+#### Each entry must have a unique name, starting with bigip/ 
+#### (e.g. bigip/1, bigip/2)
+bigip/1:
+  #### Endpoint must be specified for each device
+  #### because there's no rational default.
+  #### Set this to the management IP for the device. This must be
+  #### reachable from the Application Study Tool host.
+  endpoint: https://10.0.0.1
+  #### Override some default settings with device specific values
+  username: SOME_OVERRIDE_ACCOUNT_NAME
+  password: "${SOME_OTHER_ENV_VAR_WITH_ANOTHER_PASSWORD}"
+  #### Everything commented out here gets the value from default
+  # collection_interval: 30s
+  # data_types:
+  #   f5.dns:
+  #     enabled: false
+  #   f5.gtm:
+  #     enabled: false
+  # tls:
+  #   insecure_skip_verify: true
+  #   ca_file:
+bigip/2:
+  endpoint: https://10.0.0.2
+```
+
+
+## Configure Device Secrets
+The application study tool default configuration relies on environment variables
+which contain device access credentials. There are a number of ways to manage and
+inject secrets into a container environment (modifications to the docker-compose file
+to support your preferred management process are encouraged), but for simplicity,
+if there is a file named .env.device-secrets in the root project directory they will be
+mounted.
+
+Create a file called .env.device-secrets, and add your BIP passwords like so:
+```
+BIGIP_PASSWORD_1=foo-bar123!
+BIGIP_PASSWORD_2=bar-foo123!
+```
+
+The variable name (the part on the left of the equal sign) must match the configured
+value for the devices that use this password in config/ast_defaults.yaml or device specifc
+cofig in config/bigip_receivers.yaml. In the following example, bigip/1 uses BIGIP_PASSWORD_1
+from  the defaults and bigip/2 uses BIGIP_PASSWORD_2 from the device settings:
+
+```
+##############################
+## config/ast_defaults.yaml 
+##############################
+
+bigip_receiver_defaults:
+  ...
+  password: "${env:BIGIP_PASSWORD_1}"
+  ...
+
+##############################
+## config/bigip_receivers.yaml
+##############################
+
+# This gets the default "${env:BIGIP_PASSWORD_1}"
+bigip/1:
+  endpoint: https://10.0.0.1
+
+# This overrides it with "${env:BIGIP_PASSWORD_2}"
+bigip/2:
+  endpoint: https://10.0.0.1
+  password: ${env:BIGIP_PASSWORD_2}
+
+```
+
+## Run The Configuration Helper
+The config helper script can be run natively or via docker to merge the default and device
+level configs into the final OTEL Collector config as follows:
+```shell
+# Run the configuration generator
+docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
+```
+
+This will write 2 new files in the services/otel_collector directory:
+
+* `receivers.yaml` - The final list of scraper configs and their settings.
+* `pipelines.yaml` - The final pipeline configs that map receievers to output destinations
+(prometheus).
+
+### Configure CA File
+AST expects a valid TLS cert bundle unless `tls.insecure_skip_verify` is
+set to true for each device. In order to mount and use your CA file, you must
+configure the docker-compose.yaml file in this directory, and set the `ca_file` parameter to
+the resulting path. Example:
+
+docker-compose.yaml:
+```yaml
+  ...
+  otel-collector:
+    ...
+    volumes:
+      - ./services/otel_collector:/etc/otel-collector-config
+      - ./config/ca_bundle.pem:/etc/ssl/ca_bundle.pem
+```
+
+config/ast_defaults.yaml (or the tls section of each device in config/bigip_receivers.yaml):
+```yaml
+bigip_receiver_defaults:
+  ...
+  tls:
+    insecure_skip_verify: false
+    ca_file: "/etc/ssl/ca_bundle.pem"
+```
+
+The configuration parameter `tls.insecure_skip_verify` defaults to false. Installers
+who would like to opt-in to run in an insecure TLS mode must set
+`tls.insecure_skip_verify: true` and understand
+that the connection between the OTEL collector and the BIG-IP does not have secure
+TLS termination.
+
+### Configure Grafana
+The Grafana instance can be configured via environment variables using their standard
+[options](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables).
+
+The included .env-example can be copied over and modified to set the initial admin
+password to a value you select:
+
+```
+cp .env-example .env
+<edit .env with desired admin password and any other variables>
+```
+
+### Run Application Study Tool
+Once the above configurations have been made, the tool can be started with:
+
+```
+docker compose up
+```
+
+#### View The Dashboards
+The default Grafana user/pass is `admin/admin`, and can be accessed at
+`http://<hostname>:3000`.

pages/quickstart.md

@@ -142,10 +142,10 @@ bigip/2:
 ```
 
 ## Run The Configuration Helper
-The config helper script can be run natively or via docker to merge the default and device
-level configs into the final OTEL Collector config as follows:
+The config helper script can be run natively or via docker from the project root directory
+to merge the default and device level configs into the final OTEL Collector config as follows:
 ```shell
-# Run the configuration generator
+# Run the configuration generator from the project root directory
 docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
 ```