[Grouper/Epic] Download Functionality problems and improvements

[tatomir146]

Used internal env to test download functionality.

Describe the bug
Various things which should be improved by download functionality:

• remove "metadata" from JSON, no need to have this
• syntheticLocations are empty config: []
• yaml formatting - no new lines used, making yaml files hard readable
• yaml: quoting values makes file better readable in IDEs
• main.go: downloadSpecificAPI should be either renamed to download-specific-api to keep consistency or even better to specific-api as it's a part of download command
• main.go: p alias used for downloadSpecificApi > not really intuitiv
• main.go: help for download functionality tells Yaml file containing environment to deploy to - deploy should be renamed to download
• download.go: multiple enviroments typos
• IDs should be stripped eg: alerting-profile "mzId": "-6221876622438591000"
• verbose prints out e.g. nil as string DEBUG getting detail {... Cl_ES %!s(*string=<nil>)}
• built in extensions are also downloaded: ruxit.python.redis:slave_state:unsynchronized Redis Slave unsynchronized
• if there are multiple dashboards with same name then something like this is generated:

config:
- Home: Home.json
- Home: Home.json
- Home: Home.json
- Home: Home.json
Home:
- name: Home
- name: Home
- name: Home
- name: Home

[kristofre]

Not sure if I agree with

remove "metadata" from JSON, no need to have this

Usecase is as follows: what if I download from environment/cluster X (let's say non-prod) that is on version a and I want to apply that config to environment/cluster Y (production) that is on version b and there is a difference in API behaviour. The meta data of the version would allow us to process that properly

[ghost]

about the yaml formatting, we will have to look for an different parser from golang struct to yaml, the default one is quite strict in the convention for formatting go-yaml/yaml#556. We will have to create custom formatters to get quoting and line breaks for certain blocks.

[tatomir146]

Not sure if I agree with

remove "metadata" from JSON, no need to have this

Usecase is as follows: what if I download from environment/cluster X (let's say non-prod) that is on version a and I want to apply that config to environment/cluster Y (production) that is on version b and there is a difference in API behaviour. The meta data of the version would allow us to process that properly

But is there any value in saving this into repository? If the idea is to use download functionality to pull and store config repo ready configuration, then this configuration should also work with different cluster versions. If the main concern is validation in case of failing updates (to another environment), then there are other plans in verifying configuration compatibility, not through metada.

[cvidler]

Adding to this issue at it seems relevant, happy to raise a new issue if more appropriate.

• app-detection-rule output of the download command, only includes one rule per application, if there's more than one rule, the others are ignored - not in the .json files.

Testing on a customers environment (hoping to use monaco to migrate all config from an environment into another), 25 Web Applications, the UI reports 36 app-detection-rules, only 25 are present in the JSON files.

(using version 1.4.0).

[kristofre]

Adding to this issue at it seems relevant, happy to raise a new issue if more appropriate.

• app-detection-rule output of the download command, only includes one rule per application, if there's more than one rule, the others are ignored - not in the .json files.

Testing on a customers environment (hoping to use monaco to migrate all config from an environment into another), 25 Web Applications, the UI reports 36 app-detection-rules, only 25 are present in the JSON files.

(using version 1.4.0).

Some more meat around this:
We had a (DT Product) issue when trying to create multiple app detection rules for the same app using the API.
That was resolved, but only for newly created ones starting with 210.
But for the ones created before that we are merging them together and thus not applying them all:

config:
- Webadmintools: Webadmintools.json
- Webadmintools: Webadmintools.json
- Webadmintools: Webadmintools.json
- Webadmintools: Webadmintools.json
- Webadmintools: Webadmintools.json

Webadmintools:
- name: Web admin tools
- name: Web admin tools
- name: Web admin tools
- name: Web admin tools
- name: Web admin tools

[ghost]

another issue for the custom service download needs to remove rule id and method rule Id to allow the deploy process to be transparent for the user.

[evanderbent]

Another issue is in application-web, "identifier" is not stripped blocking a re-upload.

[UnseenWizzard]

As this issue is quite old and groups several issues, the current state needs to be checked - some of these should be fixed/about to the be fixed.
Anything not resolved will be split into it's own ticket

Current state (tick if solved already)

• remove "metadata" from JSON, no need to have this
• Download creates empty synthetic-locations config if only PUBLIC locations exists #666
  syntheticLocations are empty config: []
• WON'T FIX - non-standard yaml formatting
  Both requests deviate from standard yaml formatting and would mean rather than just marshaling structs to yaml, we would need to roll our own marshalling via templating.
  The effort for this would outweigh the user experience benefit of this
  • yaml formatting - no new lines used, making yaml files hard readable
  • yaml: quoting values makes file better readable in IDEs
• Download Command parameters are inconsistent and partially wrongly described #665
  • main.go: downloadSpecificAPI should be either renamed to download-specific-api to keep consistency or even better to specific-api as it's a part of download command
  • main.go: p alias used for downloadSpecificApi > not really intuitiv
  • main.go: help for download functionality tells Yaml file containing environment to deploy to - deploy should be renamed to download
• download.go: multiple enviroments typos
• (part of) Next download feature 2.0 #421 resolving dependencies to other downloaded configurations
  IDs should be stripped eg: alerting-profile "mzId": "-6221876622438591000"
• fix(download): Correct debug logging of config being downloaded #670
  verbose prints out e.g. nil as string DEBUG getting detail {... Cl_ES %!s(*string=)}
• Download includes built-in extensions rather than just custom ones #667
  built in extensions are also downloaded: ruxit.python.redis:slave_state:unsynchronized Redis Slave unsynchronized
• Dashboard with same name break download/deploy functionality #642
  if there are multiple dashboards with same name then something like this is generated:
• Download is not removing non-standard identifiers - custom-service #668
  the custom service download needs to remove rule id and method rule Id
• Download is not removing non-standard identifiers - application-web #669
  in application-web, "identifier" is not stripped blocking a re-upload
• Unusual indentation in downloaded config files #640
• Download is not removing non-standard indentifiers - calculated-metrics-service #178
• Downloading preset dashboards #478
• Error in deployment of synthetic-monitor with default performance thresholds #505

[UnseenWizzard]

open sub tasks to be handled with v2