[Fleet] Upgrade APIs to support moving a value from .fleet-policies to secrets index when a field moves to being secret

[juliaElastic]

Part of #154715

Scenario not implemented yet:

• Fleet has an agent policy with an integration
• Change a field in the integration to become secret in a new version (supposing feature flag is enabled)
• Upgrade the integration to use the new version
• Expect that the secret field is moved from being stored in the policy to the secrets index.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

I did a test of this by installing latest system package (1.38.2) and then uploading a newer version where made 2 existing vars secret: ignore_older in syslog datastream vars, and url in package vars in version 1.38.3.

Uploaded the package like this:
curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/julia/api/fleet/epm/packages -u elastic:changeme --data-binary @system-1.38.3.zip

system-1.38.3.zip

And then upgraded the package and integration policy from the UI.

Verified that the secret vars became secret refs, and the agent got the secret values.

GET .kibana_ingest/_search?q=ingest-package-policies.package.name:system

      "vars": {
                      "ignore_older": {
                        "type": "text",
                        "value": {
                          "isSecretRef": true,
                          "id": "6LN09IkB4EEjyKF9cMBH"
                        }
                      },

  "url": {
                    "type": "text",
                    "value": {
                      "isSecretRef": true,
                      "id": "6bN09IkB4EEjyKF9cMBH"
                    }
                  },

So I think the main functionality works as expected.

One thing I'm not sure is a bug or expected, is that when I look at the agent policy with View policy, I see the request.url with the secret envvar, having a path after it:

When I look at the same config in agent computed-config.yaml, I only see request.url: https://server.example.com:8090 that is coming from the secret value. So I'm not sure if the path /services/search/jobs/export should be there or not.

EDIT: checked the same variables without secrets (system-1.38.2) and the agent gets the full url, so I think this is a bug in fleet-server logic that replaces the variables:
should be: request.url: https://server.example.com:8089/services/search/jobs/export

EDIT: Raised a pr to do some cleanup elastic/fleet-server#2876 and can't reproduce the request.url issue anymore locally, the computed config contains the path after the secret value as expected.
request.url: https://server.example.com:8090/services/search/jobs/export

[joshdover]

@juliaElastic Doesn't this show that the upgrade scenario in this issue is already working?

I think we should have explicit API functional tests for this in any case.

[juliaElastic]

@joshdover Yes, I think this is working.
Hey @jillguyonnet, you could add some API integration tests for this scenario in this issue, as it looks like the implementation is already done.

[jlind23]

@jillguyonnet can you add your integration tests PR here as soon as ready? Thanks