Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TSDB] Dimension limit override settings blocking the build #1174

Closed
agithomas opened this issue Mar 6, 2023 · 25 comments
Closed

[TSDB] Dimension limit override settings blocking the build #1174

agithomas opened this issue Mar 6, 2023 · 25 comments

Comments

@agithomas
Copy link
Contributor

agithomas commented Mar 6, 2023

Refer : https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-5312/12/pipeline

The configuration to override the default limit is provided as

elasticsearch:
  index_template:
    settings:
      index.mapping.dimension_fields.limit: 32

Inspite this is a valid configuration, the integration test error appears. This may have an impact on more than one packages , not just limited to the influxdb.

@agithomas agithomas changed the title [TSDB] Dimension limit blocking the build [TSDB] Dimension limit override settings blocking the build Mar 6, 2023
@jsoriano
Copy link
Member

jsoriano commented Mar 6, 2023

@agithomas could you try to install the package and check in the index templates if the setting is there?

@agithomas
Copy link
Contributor Author

@agithomas could you try to install the package and check in the index templates if the setting is there?

Yes, i can find that in the index template.

    "mapping": {
      "total_fields": {
        "limit": "10000"
      },
      "dimension_fields": {
        "limit": "32"
      }
    },

@jsoriano
Copy link
Member

jsoriano commented Mar 6, 2023

Is this in the main/composed index template, or in the component template with the package fields? If it is in the main one maybe it is applying the limits per component template 🤔

@jsoriano
Copy link
Member

jsoriano commented Mar 6, 2023

Looks related: elastic/elasticsearch#93564

@jsoriano
Copy link
Member

jsoriano commented Mar 6, 2023

Is this in the main/composed index template, or in the component template with the package fields? If it is in the main one maybe it is applying the limits per component template thinking

It is in the same component template where the package fields are defined.

@martijnvg is this expected? Should we install the dimension fields limit in the composed index template?

Find here a dump of the templates installed for this package: package-dump.zip

@agithomas
Copy link
Contributor Author

agithomas commented Mar 7, 2023

I think, i understood what is happening.

  • The above issue is found when i tried to enable TSDB for influxdb package, having 8.3.0 as the minimum kibana version also the ES version.

  • When the test is run, 8.3.0 ES is created and the tests are executed on 8.3.0 stack.

  • In 8.3.0, it appears, 16 is the default dimension limit and it cannot be overwritten by the settings such as

elasticsearch:
  index_template:
    settings:
      index.mapping.dimension_fields.limit: 32

@martijnvg , from which version of the ES, the ability to override the dimension limit (ie: index.mapping.dimension_fields.limit: <new limit> )exist in ES?

@martijnvg
Copy link
Member

What is the actual error that occurs here? This particular setting does seem to exist in 8.3.0 and earlier versions.

However note that TSDB became available as tech preview (elastic/elasticsearch#90116)) with version 8.5.0 and many other tsdb related settings were not available before version 8.5.0.

@agithomas
Copy link
Contributor Author

What is the actual error that occurs here? This particular setting does seem to exist in 8.3.0 and earlier versions.

[illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded"

@martijnvg
Copy link
Member

When does this error occur? When adding a component template, index template or when a data stream gets created?

@agithomas
Copy link
Contributor Author

When does this error occur? When adding a component template, index template or when a data stream gets created?

having a matching regex `influx', in kibana. But, i do not find anything that match.

So, i can't really tell the stage at where the failure occur in 8.3.0 stack.

Any other steps i can run to get the exact stage of failure?

image

@agithomas
Copy link
Contributor Author

When does this error occur? When adding a component template, index template or when a data stream gets created?

The extended error log line from the integration test log is as below

Error: error running package system tests: could not complete test run: could not add data stream config to policy: could not add package to policy; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"Error installing influxdb 0.3.0: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded"}

@lalit-satapathy
Copy link

lalit-satapathy commented Mar 7, 2023

When the test is run, 8.3.0 ES is created and the tests are executed on 8.3.0 stack.

The issue could be because of "system test" is running an older version of the ES.

@martijnvg
Copy link
Member

I tried installing the templates in the package dump here locally and everything seems to work ok. No validation errors.
The index.mapping.dimension_fields.limit index setting exists for a long time already (from 8.0 if I'm not mistaken) and has always been a dynamic setting, meaning that it could be updated.

The mentioned Limit of total dimension fields [16] has been exceeded error indicates that somehow limit has not been increased to 48 or 32 which both component templates are doing. It would be helpful to know which Elasticsearch api call returns this error and then check the associated data stream or template. From what I can see this shouldn't occur if any of the templates in the package dump get installed.

@agithomas
Copy link
Contributor Author

Testing with 8.6 stack . Gets a successful run of system test

image

Testing with 8.4 Stack . Do not get a successful run of system test

image

** Testing with 8.5 Stack. Do not get a successful run of the system test **

image

** Tested again with 8.6 Stack. Gets a successful run **

image

** Tested with 8.7.0-Snapshot. Gets a successful run **

image

In all the above tests, only one thing that differs in the ES stack version.

Are two different APIs getting invoked before and after 8.6.0 ? @ruflin , @jsoriano , do you know? Or any means of knowing / verifying this?

@jsoriano
Copy link
Member

jsoriano commented Mar 7, 2023

The failure happens in the request to Kibana to add the package policy (POST kbn:/api/fleet/fleet_policies). It can be also reproduced when installing a package using POST kbn:/api/fleet/epm/packages/influxdb-0.3.0.

I don't know exactly on what request to ES it fails. It only logs:

[2023-03-07T19:08:54.844+00:00][WARN ][plugins.fleet] Failure to install package [influxdb]: [ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded]

The dump I did in #1174 (comment) was with 8.6.1, actually if I try to dump on previous versions I cannot because the package cannot be installed.

In principle elastic-package is doing exactly the same requests in 8.5 and 8.6, @martijnvg could it be that something related has changed on how these settings are handled in 8.6?

Or @kpollich, could it be that something related changed in how Kibana installs packages in 8.6?

@agithomas the quickest path for you to unblock your progress will probably be to update the kibana version constraint to ^8.6.0 if this is an option.

@lalit-satapathy
Copy link

The kibana.version issue was discussed as a feedback from @ruflin earlier.

Ideal behaviour for us that index.mapping.dimension_fields.limit check is ignored on older versions and not throw an error. Is that feasible?

@martijnvg
Copy link
Member

In principle elastic-package is doing exactly the same requests in 8.5 and 8.6, @martijnvg could it be that something related has changed on how these settings are handled in 8.6?

Not that I know of. The mentioned setting (index.mapping.dimension_fields.limit) and the ability to define fields as dimensions hasn't changed to my knowledge in the mentioned versions.

@kpollich
Copy link
Member

kpollich commented Mar 8, 2023

[2023-03-07T19:08:54.844+00:00][WARN ][plugins.fleet] Failure to install package [influxdb]: [ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded]

Ideal behaviour for us that index.mapping.dimension_fields.limit check is ignored on older versions and not throw an error. Is that feasible?

Or @kpollich, could it be that something related changed in how Kibana installs packages in 8.6?

Docs for this setting are here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit

Fleet does not provide this setting in any way, so we're just getting the default value of 16 here. If we're exceeding the limit then I view this as an expected error. We'd need a way for the package to export a different limit or for Fleet to hardcode a higher one into all index templates.

As for why this fails in 8.6 - Fleet didn't support TSDB in any way prior to 8.6. As of elastic/kibana#144974, Fleet actually honors dimension fields and enables time series indexing on appropriate data streams. So, in earlier versions all time series metrics, dimensions, etc are ignored when installing the package. In 8.6, Fleet will actually honor these settings.

@jsoriano
Copy link
Member

Docs for this setting are here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit

Fleet does not provide this setting in any way, so we're just getting the default value of 16 here. If we're exceeding the limit then I view this as an expected error. We'd need a way for the package to export a different limit or for Fleet to hardcode a higher one into all index templates.

Packages can already override this limit through elasticsearch.index_template.settings, as is the case here (from this PR). And Fleet is installing it (at least from 8.6), as seen on the dump on this comment #1174 (comment).

As for why this fails in 8.6 - Fleet didn't support TSDB in any way prior to 8.6. As of elastic/kibana#144974, Fleet actually honors dimension fields and enables time series indexing on appropriate data streams. So, in earlier versions all time series metrics, dimensions, etc are ignored when installing the package. In 8.6, Fleet will actually honor these settings.

It seems that dimensions are installed before 8.6, even if TSDB is not enabled yet. The error about dimensions limit exceeded appears on previous versions.

@martijnvg could it be that dimension_fields.limit setting is only leveraged if tsdb mode is enabled?

@lalit-satapathy
Copy link

Need a closure on this. Although currently only InfluxDB is impacted, but that package is already on 8.6. We will see this issue impacting more packages once they start migrating.

@jsoriano
Copy link
Member

jsoriano commented Apr 4, 2023

I am afraid that the only solution at this point is to increase the kibana version constraint to at least ^8.6.0 for packages that need to increase the dimensions limit.
The alternative would be to change the behaviour in older versions of kibana and/or elasticsearch, and I don't think this is feasible at this point 😕

@ruflin
Copy link
Member

ruflin commented Apr 4, 2023

@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit?

@lalit-satapathy
Copy link

@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit?

InfluxDB and may be few other prometheus type packages. The current understanding is, we may force kibana.version for TSDB packages >= 8.8. as per that plan, this issue may not be considered critical.

@lalit-satapathy
Copy link

@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit?

InfluxDB and may be few other prometheus type packages. The current understanding is, we may force kibana.version for TSDB packages >= 8.8. as per that plan, this issue may not be considered critical.

@agithomas, Should we keep this open?

@agithomas
Copy link
Contributor Author

Closing this because it is decided that 8.8.x will be the min version for TSDB enablement for integration packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants