add stats api #3
Conversation
| constructor(private response: KibanaResponseFactory) {} | ||
|
|
||
| error<T>({ statusCode, body, headers }: CustomHttpResponseOptions<T>) { | ||
| // KibanaResponse is not exported so we cannot use a return type here and that is why the linter is turned off above |
There was a problem hiding this comment.
consider removing this comment if it's no longer applicable
There was a problem hiding this comment.
when running the following command:
node scripts/type_check.js --project x-pack/plugins/data_quality/tsconfig.jsonthe following type error is reported:
[bazel] packages/kbn-securitysolution-es-utils/src/build_response/index.ts:9:71 - error TS2307: Cannot find module '@kbn/core/server' or its corresponding type declarations.
[bazel]
[bazel] 9 import type { CustomHttpResponseOptions, KibanaResponseFactory } from '@kbn/core/server';
[bazel] ~~~~~~~~~~~~~~~~~~
[bazel]
[bazel] Found 1 error in packages/kbn-securitysolution-es-utils/src/build_response/index.ts:9
| path: GET_INDEX_MAPPINGS, | ||
| validate: { | ||
| params: schema.object({ | ||
| index_name: schema.string(), |
There was a problem hiding this comment.
Our current plan (in the Data Quality dashboard) is to only request a specific index name, e.g. .ds-logs-endpoint.alerts-default-2022.11.17-000001 from this route, but given that the underlying ES mappings API also accepts a pattern, e.g. .ds-logs-*, consider renaming the request.params.index_name parameter to request.params.pattern, to align the parameter name with the full capabilities of the API.
| path: GET_INDEX_STATS, | ||
| validate: { | ||
| params: schema.object({ | ||
| index_name: schema.string(), |
There was a problem hiding this comment.
Consider renaming the request.params.index_name parameter to request.params.pattern, because the Data Quality Dashboard will provide (as input to this route) a pattern like logs-*, to enumerate all the indexes that match the specified pattern.
| const { client } = (await context.core).elasticsearch; | ||
| const decodedIndexName = decodeURIComponent(request.params.index_name); | ||
|
|
||
| const mappings = await fetchStats(client, decodedIndexName); |
There was a problem hiding this comment.
consider renaming mappings to stats
| const mappings = await fetchStats(client, decodedIndexName); | ||
|
|
||
| return response.ok({ | ||
| body: mappings.indices, |
There was a problem hiding this comment.
The body returned here is of type Record<string, IndicesStatsIndicesStats>. The IndicesStatsIndicesStats type may be imported via:
import type { IndicesStatsIndicesStats } from '@elastic/elasticsearch/lib/api/types';Would it be reasonable to use io-ts here to verify the shape of the API response matches the type above?
If yes, would it be reasonable to create a common io-ts validation code that could be used both here (in the server), and on the client?
| const mappings = await fetchMappings(client, decodedIndexName); | ||
|
|
||
| return response.ok({ | ||
| body: mappings, |
There was a problem hiding this comment.
The body returned here is of type Record<string, IndicesGetMappingIndexMappingRecord>. The IndicesGetMappingIndexMappingRecord type may be imported via:
import type { IndicesGetMappingIndexMappingRecord } from '@elastic/elasticsearch/lib/api/types';Would it be reasonable to use io-ts here to verify the shape of the API response matches the type above?
If yes, would it be reasonable to create a common io-ts validation code that could be used both here (in the server), and on the client?
There was a problem hiding this comment.
Thanks @angorayc for implementing these APIs 🙏
Running a local copy of this branch, I integrated with both the stats and mappings APIs, and everything worked perfectly!
LGTM 🚀
|
Thanks for the feedback Andrew @andrew-goldstein , I've done the renaming and added some basic input validation with io-ts. I haven't done the io-ts validation for api responses, is it ok if I do that in a following pr? The other issue I have is the type issue, somehow the type check just doesn't like this import, but this is what other places doing. I'm still trying to figure this out. |
|
Update: After adding |
## Summary Related to elastic#144161 Found that on a bulk update tags task failure, the task didn't stop after 3 retries (should be over in less then a minute), the retries kept happening for 2 hours. This change removes the retry task if 3 retries are reached. Also testing in cloud deployment to see if the tags error can be reproduced with this fix. I could reproduce the reported error locally, and seeing it goes away with this fix. To verify: - Add at least 50k agents with the `create_agents` script in kibana repo - open Kibana, select the 50k agents, and open Actions / Add tags - Try this in a few seconds: add 2 new tags, and remove one of them - Wait about 30s, the agents should reflect the changes - Check the logs to see that the tasks are removed after 3rd retry is reached or successful. - Check that there are no more running tasks. Any running task can be found in Kibana Console by running this query: `GET .kibana_task_manager/_search?q=task.taskType:"fleet:update_agent_tags:retry"` Locally simulated an error to test that the retry (and check) task is removed: ``` [2022-12-07T15:52:16.415+01:00][ERROR][plugins.fleet] Retry #3 of task fleet:update_agent_tags:retry:848984ab-c11d-4ebe-8d1f-606143dd656b failed: failing task [2022-12-07T15:52:16.416+01:00][WARN ][plugins.fleet] Stopping after 3rd retry. Error: failing task [2022-12-07T15:52:16.416+01:00][INFO ][plugins.fleet] Removing task fleet:update_agent_tags:retry:check:848984ab-c11d-4ebe-8d1f-606143dd656b [2022-12-07T15:52:16.416+01:00][INFO ][plugins.fleet] Removing task fleet:update_agent_tags:retry:848984ab-c11d-4ebe-8d1f-606143dd656b ```
## Summary Resolves elastic#143905. This PR adds support for integration-level outputs. This means that different integrations within the same agent policy can now be configured to send data to different locations. This feature is gated behind `enterprise` level subscription. For each input, the agent policy will configure sending data to the following outputs in decreasing order of priority: 1. Output set specifically on the integration policy 2. Output set specifically on the integration's parent agent policy (including the case where an integration policy belongs to multiple agent policies) 3. Global default data output set via Fleet Settings Integration-level outputs will respect the same rules as agent policy-level outputs: - Certain integrations are disallowed from using certain output types, attempting to add them to each other via creation, updating, or "defaulting", will fail - `fleet-server`, `synthetics`, and `apm` can only use same-cluster Elasticsearch output - When an output is deleted, any integrations that were specifically using it will "clear" their output configuration and revert back to either `#2` or `#3` in the above list - When an output is edited, all agent policies across all spaces that use it will be bumped to a new revision, this includes: - Agent policies that have that output specifically set in their settings (existing behavior) - Agent policies that contain integrations which specifically has that output set (new behavior) - When a proxy is edited, the same new revision bump above will apply for any outputs using that proxy The final agent policy YAML that is generated will have: - `outputs` block that includes: - Data and monitoring outputs set at the agent policy level (existing behavior) - Any additional outputs set at the integration level, if they differ from the above - `outputs_permissions` block that includes permissions for each Elasticsearch output depending on which integrations and/or agent monitoring are assigned to it Integration policies table now includes `Output` column. If the output is defaulting to agent policy-level output, or global setting output, a tooltip is shown: <img width="1392" alt="image" src="https://github.com/user-attachments/assets/5534716b-49b5-402a-aa4a-4ba6533e0ca8"> Configuring an integration-level output is done under Advanced options in the policy editor. Setting to the blank value will "clear" the output configuration. The list of available outputs is filtered by what outputs are available for that integration (see above): <img width="799" alt="image" src="https://github.com/user-attachments/assets/617af6f4-e8f8-40b1-b476-848f8ac96e76"> An example of failure: ES output cannot be changed to Kafka while there is an integration <img width="1289" alt="image" src="https://github.com/user-attachments/assets/11847eb5-fd5d-4271-8464-983d7ab39218"> ## TODO - [x] Adjust side effects of editing/deleting output when policies use it across different spaces - [x] Add API integration tests - [x] Update OpenAPI spec - [x] Create doc issue ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>
Summary
This is a part of https://github.com/elastic/security-team/issues/5578
Request:
example: GET /internal/data_quality/stats/auditbeat-*
Response: