-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TSDB] Dimension limit override settings blocking the build #1174
Comments
@agithomas could you try to install the package and check in the index templates if the setting is there? |
Yes, i can find that in the index template.
|
Is this in the main/composed index template, or in the component template with the package fields? If it is in the main one maybe it is applying the limits per component template 🤔 |
Looks related: elastic/elasticsearch#93564 |
It is in the same component template where the package fields are defined. @martijnvg is this expected? Should we install the dimension fields limit in the composed index template? Find here a dump of the templates installed for this package: package-dump.zip |
I think, i understood what is happening.
@martijnvg , from which version of the ES, the ability to override the dimension limit (ie: |
What is the actual error that occurs here? This particular setting does seem to exist in 8.3.0 and earlier versions. However note that TSDB became available as tech preview (elastic/elasticsearch#90116)) with version 8.5.0 and many other tsdb related settings were not available before version 8.5.0. |
[illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded" |
When does this error occur? When adding a component template, index template or when a data stream gets created? |
having a matching regex `influx', in kibana. But, i do not find anything that match. So, i can't really tell the stage at where the failure occur in 8.3.0 stack. Any other steps i can run to get the exact stage of failure? |
The extended error log line from the integration test log is as below
|
The issue could be because of "system test" is running an older version of the ES. |
I tried installing the templates in the package dump here locally and everything seems to work ok. No validation errors. The mentioned |
Testing with 8.6 stack . Gets a successful run of system test Testing with 8.4 Stack . Do not get a successful run of system test ** Testing with 8.5 Stack. Do not get a successful run of the system test ** ** Tested again with 8.6 Stack. Gets a successful run ** ** Tested with 8.7.0-Snapshot. Gets a successful run ** In all the above tests, only one thing that differs in the ES stack version. Are two different APIs getting invoked before and after 8.6.0 ? @ruflin , @jsoriano , do you know? Or any means of knowing / verifying this? |
The failure happens in the request to Kibana to add the package policy ( I don't know exactly on what request to ES it fails. It only logs:
The dump I did in #1174 (comment) was with 8.6.1, actually if I try to dump on previous versions I cannot because the package cannot be installed. In principle Or @kpollich, could it be that something related changed in how Kibana installs packages in 8.6? @agithomas the quickest path for you to unblock your progress will probably be to update the kibana version constraint to |
Not that I know of. The mentioned setting ( |
Docs for this setting are here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#mapping-settings-limit Fleet does not provide this setting in any way, so we're just getting the default value of As for why this fails in 8.6 - Fleet didn't support TSDB in any way prior to 8.6. As of elastic/kibana#144974, Fleet actually honors dimension fields and enables time series indexing on appropriate data streams. So, in earlier versions all time series metrics, dimensions, etc are ignored when installing the package. In 8.6, Fleet will actually honor these settings. |
Packages can already override this limit through
It seems that dimensions are installed before 8.6, even if TSDB is not enabled yet. The error about dimensions limit exceeded appears on previous versions. @martijnvg could it be that |
Need a closure on this. Although currently only InfluxDB is impacted, but that package is already on 8.6. We will see this issue impacting more packages once they start migrating. |
I am afraid that the only solution at this point is to increase the kibana version constraint to at least |
@lalit-satapathy How many packages are affected at the moment by requiring going above the default limit? |
InfluxDB and may be few other prometheus type packages. The current understanding is, we may force kibana.version for TSDB packages >= 8.8. as per that plan, this issue may not be considered critical. |
@agithomas, Should we keep this open? |
Closing this because it is decided that 8.8.x will be the min version for TSDB enablement for integration packages. |
Refer : https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-5312/12/pipeline
The configuration to override the default limit is provided as
Inspite this is a valid configuration, the integration test error appears. This may have an impact on more than one packages , not just limited to the influxdb.
The text was updated successfully, but these errors were encountered: