Can't upgrade APM integration to version 7.16.1 on ESS

[bmorelli25]

APM Server version (apm-server version):

• ESS version 7.16.1
• APM integration version 7.16.0

Description of the problem including expected versus actual behavior:

• Can't upgrade the APM integration to 7.16.1
• Can't save configuration options

Steps to reproduce:

1. Spin up 7.16.1 on ESS
2. Click Add integration > Elastic APM > APM integration > Switch to Elastic Agent > Switch to Elastic Agent. You should now have version 7.16.0 of the APM integration installed.
3. Select View the APM integration in Fleet > Actions > Edit integration
4. Click Upgrade integration. This is the error I get:
   
   I expanded the "Advanced options" and saw "Maximum duration for reading an entire request" thinks the default 3600 is less than 1ms, so I took that down to 0 (which removes the red) and tried again. Same error.

I think it's important to note that I didn't change any configs. I was just trying to get the URL and Host from this page.

[simitt]

I just confirmed that this also happens in 7.16.0.

[simitt]

This also happens after adding a default APM integration to any agent policy and trying to upgrade it. This looks like an issue with the APM UI editor rather than an APM Server issue, therefore transferring the issue.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dannycroft]

@bmorelli25 thank you for raising this, we'll take a look at it ASAP (the team are coming to the end of thier on week)

[gbamparop]

The form validation error for "Maximum duration for reading an entire request" happens because ms is missing. For example, 3600ms is valid. This doesn't fix the other error (definition for this key is missing) when submitting the form though, which is caused by the elasticsearch property being sent

[gbamparop]

Fleet allows packages to specify cluster privileges as part of this but the update policy endpoint throws a Bad Request error when adding this param:

"elasticsearch": {
        "privileges": {
            "cluster": [
                "cluster:monitor/main"
            ]
        }
    }

[joshdover]

FYI @hop-dev is taking a look at this validation issue ^

[joshdover]

@ogupte This looks suspiciously similar to the fix you made in this PR for 8.0 and 8.1: #119131 (comment)

Long story short is that we don't accept this parameter on the API, it's supplied by the package manifest and added to the package policy on the backend. I think we need to make a similar adjustment on 7.16 branch to not send in this API request.

[gbamparop]

@joshdover it seems the same issue

[joshdover]

AFAIK this has been fixed. Still waiting for someone from @elastic/apm-ui to confirm before we close. We also need to figure out why this wasn't caught in testing.

@simitt is it possible this wasn't caught because we aren't testing against production packages in the staging environment? I've been worried that that could cause issues like this that aren't caught until we promote a release to production.

[sorenlouv]

@cauemarcondes Can you confirm if this is fixed?

[simitt]

@simitt is it possible this wasn't caught because we aren't testing against production packages in the staging environment? I've been worried that that could cause issues like this that aren't caught until we promote a release to production.

We werent testing against production packages, but we were testing against the production like packages from staging, that then got promoted to production. I'm not sure I am following how this would have caused troubles here.

[cauemarcondes]

@joshdover @hop-dev I see that this PR #121412 hasn't been backported to 8.0, is it correct?

[joshdover]

@cauemarcondes that change was already present in #119131 which was merged to 8.1 and 8.0

[cauemarcondes]

@sqren I tested on 8.0 and I couldn't reproduce the error.

[joshdover]

Think we're good here, closing.

[marclop]

I'm still seeing this error when trying to upgrade the APM Integration using Kibana 7.17.0. I can reproduce locally and using the latest 7.17.0 build in ESS. Was the fix not back ported to the 7.17 branch? Exact steps to reproduce:

1. Create a deployment using version 7.17.0 with APM & Fleet server enabled
2. Install the APM Integration (7.16.0)
3. Upgrade the APM Integration to 7.16.1
4. When saving the changes, you'll see the following message:

[cauemarcondes]

@hop-dev can you check if your fix was backported to 7.17?

[ogupte]

@hop-dev can you check if your fix was backported to 7.17?

I don't see it in 7.17.

[ogupte]

backport created for 7.17: #123368

[marclop]

Thank you @ogupte !