From d1336217bdfa5818d7d75d2fb45aae1d28a6ff21 Mon Sep 17 00:00:00 2001 From: Zachary Tong Date: Wed, 13 Jun 2018 14:21:31 +0000 Subject: [PATCH] [Docs] All Rollup docs experimental, better agg limitations, clarify DeleteJob - All rollup pages should be marked as experimental instead of just the top page - While the job config docs state which aggregations are allowed, adding a section which specifically details this in one place is more convenient for the user - Add a clarification that the DeleteJob API does not delete the rollup data, just the rollup job. --- .../en/rest-api/rollup/delete-job.asciidoc | 28 +++++++++++++++++++ .../docs/en/rest-api/rollup/get-job.asciidoc | 2 ++ .../docs/en/rest-api/rollup/put-job.asciidoc | 2 ++ .../en/rest-api/rollup/rollup-caps.asciidoc | 2 ++ .../rollup/rollup-job-config.asciidoc | 2 ++ .../en/rest-api/rollup/rollup-search.asciidoc | 2 ++ .../en/rest-api/rollup/start-job.asciidoc | 2 ++ .../docs/en/rest-api/rollup/stop-job.asciidoc | 2 ++ x-pack/docs/en/rollup/api-quickref.asciidoc | 2 ++ x-pack/docs/en/rollup/index.asciidoc | 4 ++- x-pack/docs/en/rollup/overview.asciidoc | 2 ++ .../en/rollup/rollup-agg-limitations.asciidoc | 24 ++++++++++++++++ .../en/rollup/rollup-getting-started.asciidoc | 2 ++ .../rollup/rollup-search-limitations.asciidoc | 6 ++-- .../en/rollup/understanding-groups.asciidoc | 2 ++ 15 files changed, 81 insertions(+), 3 deletions(-) create mode 100644 x-pack/docs/en/rollup/rollup-agg-limitations.asciidoc diff --git a/x-pack/docs/en/rest-api/rollup/delete-job.asciidoc b/x-pack/docs/en/rest-api/rollup/delete-job.asciidoc index 056a4470480a0..b795e0b28c760 100644 --- a/x-pack/docs/en/rest-api/rollup/delete-job.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/delete-job.asciidoc @@ -5,9 +5,37 @@ Delete Job ++++ +experimental[] + This API deletes an existing rollup job. The job can be started or stopped, in both cases it will be deleted. Attempting to delete a non-existing job will throw an exception +.Deleting the job does not delete rolled up data +********************************** +When a job is deleted, that only removes the process that is actively monitoring and rolling up data. +It does not delete any previously rolled up data. This is by design; a user may wish to roll up a static dataset. Because +the dataset is static, once it has been fully rolled up there is no need to keep the indexing Rollup job around (as there +will be no new data). So the job may be deleted, leaving behind the rolled up data for analysis. + +If you wish to also remove the rollup data, and the rollup index only contains the data for a single job, you can simply +delete the whole rollup index. If the rollup index stores data from several jobs, you must issue a Delete-By-Query that +targets the Rollup job's ID in the rollup index: + + +[source,js] +-------------------------------------------------- +POST my_rollup_index/_delete_by_query +{ + "query": { + "term": { + "_rollup.id": "the_rollup_job_id" + } + } +} +-------------------------------------------------- +// NOTCONSOLE + +********************************** ==== Request `DELETE _xpack/rollup/job/` diff --git a/x-pack/docs/en/rest-api/rollup/get-job.asciidoc b/x-pack/docs/en/rest-api/rollup/get-job.asciidoc index 4482a87527930..7a7db9258b88a 100644 --- a/x-pack/docs/en/rest-api/rollup/get-job.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/get-job.asciidoc @@ -5,6 +5,8 @@ Get Job ++++ +experimental[] + This API returns the configuration, stats and status of rollup jobs. The API can return the details for a single job, or for all jobs. diff --git a/x-pack/docs/en/rest-api/rollup/put-job.asciidoc b/x-pack/docs/en/rest-api/rollup/put-job.asciidoc index 2cc869e1e3467..1449acadc636d 100644 --- a/x-pack/docs/en/rest-api/rollup/put-job.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/put-job.asciidoc @@ -5,6 +5,8 @@ Create Job ++++ +experimental[] + This API enables you to create a rollup job. The job will be created in a `STOPPED` state, and must be started with the <>. diff --git a/x-pack/docs/en/rest-api/rollup/rollup-caps.asciidoc b/x-pack/docs/en/rest-api/rollup/rollup-caps.asciidoc index 5a4dab69d937f..270ad005144ac 100644 --- a/x-pack/docs/en/rest-api/rollup/rollup-caps.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/rollup-caps.asciidoc @@ -5,6 +5,8 @@ Get Rollup Caps ++++ +experimental[] + This API returns the rollup capabilities that have been configured for an index or index pattern. This API is useful because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only certain aggregations can be configured for various fields, leading to a limited subset of functionality depending on diff --git a/x-pack/docs/en/rest-api/rollup/rollup-job-config.asciidoc b/x-pack/docs/en/rest-api/rollup/rollup-job-config.asciidoc index 85f1a57caa763..ef0ea6f00f7ce 100644 --- a/x-pack/docs/en/rest-api/rollup/rollup-job-config.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/rollup-job-config.asciidoc @@ -2,6 +2,8 @@ [[rollup-job-config]] === Rollup Job Configuration +experimental[] + The Rollup Job Configuration contains all the details about how the rollup job should run, when it indexes documents, and what future queries will be able to execute against the rollup index. diff --git a/x-pack/docs/en/rest-api/rollup/rollup-search.asciidoc b/x-pack/docs/en/rest-api/rollup/rollup-search.asciidoc index 557953fefb231..470cbc4eaf57d 100644 --- a/x-pack/docs/en/rest-api/rollup/rollup-search.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/rollup-search.asciidoc @@ -5,6 +5,8 @@ Rollup Search ++++ +experimental[] + The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint is needed because, internally, rolled-up documents utilize a different document structure than the original data. The Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response diff --git a/x-pack/docs/en/rest-api/rollup/start-job.asciidoc b/x-pack/docs/en/rest-api/rollup/start-job.asciidoc index b8eccd5fbce82..9a0a0a7e4f01c 100644 --- a/x-pack/docs/en/rest-api/rollup/start-job.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/start-job.asciidoc @@ -5,6 +5,8 @@ Start Job ++++ +experimental[] + This API starts an existing, stopped rollup job. If the job does not exist an exception will be thrown. Starting an already started job has no action. diff --git a/x-pack/docs/en/rest-api/rollup/stop-job.asciidoc b/x-pack/docs/en/rest-api/rollup/stop-job.asciidoc index 9da3872a10b00..6050740270503 100644 --- a/x-pack/docs/en/rest-api/rollup/stop-job.asciidoc +++ b/x-pack/docs/en/rest-api/rollup/stop-job.asciidoc @@ -5,6 +5,8 @@ Stop Job ++++ +experimental[] + This API stops an existing, started rollup job. If the job does not exist an exception will be thrown. Stopping an already stopped job has no action. diff --git a/x-pack/docs/en/rollup/api-quickref.asciidoc b/x-pack/docs/en/rollup/api-quickref.asciidoc index 1ae6de4ee011c..10aed1b572d38 100644 --- a/x-pack/docs/en/rollup/api-quickref.asciidoc +++ b/x-pack/docs/en/rollup/api-quickref.asciidoc @@ -1,6 +1,8 @@ [[rollup-api-quickref]] == API Quick Reference +experimental[] + Most {rollup} endpoints have the following base: [source,js] diff --git a/x-pack/docs/en/rollup/index.asciidoc b/x-pack/docs/en/rollup/index.asciidoc index 69cd872e59669..9ac89341bfe99 100644 --- a/x-pack/docs/en/rollup/index.asciidoc +++ b/x-pack/docs/en/rollup/index.asciidoc @@ -18,7 +18,8 @@ for analysis, but at a fraction of the storage cost of raw data. * <> * <> * <> -* <> +* <> +* <> -- @@ -27,4 +28,5 @@ include::overview.asciidoc[] include::api-quickref.asciidoc[] include::rollup-getting-started.asciidoc[] include::understanding-groups.asciidoc[] +include::rollup-agg-limitations.asciidoc[] include::rollup-search-limitations.asciidoc[] \ No newline at end of file diff --git a/x-pack/docs/en/rollup/overview.asciidoc b/x-pack/docs/en/rollup/overview.asciidoc index cee244a2ec241..a3f29f23bd107 100644 --- a/x-pack/docs/en/rollup/overview.asciidoc +++ b/x-pack/docs/en/rollup/overview.asciidoc @@ -1,6 +1,8 @@ [[rollup-overview]] == Overview +experimental[] + Time-based data (documents that are predominantly identified by their timestamp) often have associated retention policies to manage data growth. For example, your system may be generating 500,000 documents every second. That will generate 43 million documents per day, and nearly 16 billion documents a year. diff --git a/x-pack/docs/en/rollup/rollup-agg-limitations.asciidoc b/x-pack/docs/en/rollup/rollup-agg-limitations.asciidoc new file mode 100644 index 0000000000000..cd20622d93c8d --- /dev/null +++ b/x-pack/docs/en/rollup/rollup-agg-limitations.asciidoc @@ -0,0 +1,24 @@ +[[rollup-agg-limitations]] +== Rollup Aggregation Limitations + +experimental[] + +There are some limitations to how fields can be rolled up / aggregated. This page highlights the major limitations so that +you are aware of them. + +[float] +=== Limited aggregation components + +The Rollup functionality allows fields to be grouped with the following aggregations: + +- Date Histogram aggregation +- Histogram aggregation +- Terms aggregation + +And the following metrics are allowed to be specified for numeric fields: + +- Min aggregation +- Max aggregation +- Sum aggregation +- Average aggregation +- Value Count aggregation \ No newline at end of file diff --git a/x-pack/docs/en/rollup/rollup-getting-started.asciidoc b/x-pack/docs/en/rollup/rollup-getting-started.asciidoc index cf96d67454083..24f68dddd8101 100644 --- a/x-pack/docs/en/rollup/rollup-getting-started.asciidoc +++ b/x-pack/docs/en/rollup/rollup-getting-started.asciidoc @@ -1,6 +1,8 @@ [[rollup-getting-started]] == Getting Started +experimental[] + To use the Rollup feature, you need to create one or more "Rollup Jobs". These jobs run continuously in the background and rollup the index or indices that you specify, placing the rolled documents in a secondary index (also of your choosing). diff --git a/x-pack/docs/en/rollup/rollup-search-limitations.asciidoc b/x-pack/docs/en/rollup/rollup-search-limitations.asciidoc index de47404a29da3..57ba23eebccbe 100644 --- a/x-pack/docs/en/rollup/rollup-search-limitations.asciidoc +++ b/x-pack/docs/en/rollup/rollup-search-limitations.asciidoc @@ -1,6 +1,8 @@ [[rollup-search-limitations]] == Rollup Search Limitations +experimental[] + While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once live data is thrown away, you will always lose some flexibility. @@ -100,8 +102,8 @@ The Rollup functionality allows `query`'s in the search request, but with a limi - MatchAll Query - Any compound query (Boolean, Boosting, ConstantScore, etc) -Furthermore, these queries can only use fields that were also saved in the rollup job. If you wish to filter on a keyword `hostname` field, -that field must have been configured in the rollup job under a `terms` grouping. +Furthermore, these queries can only use fields that were also saved in the rollup job as a `group`. +If you wish to filter on a keyword `hostname` field, that field must have been configured in the rollup job under a `terms` grouping. If you attempt to use an unsupported query, or the query references a field that wasn't configured in the rollup job, an exception will be thrown. We expect the list of support queries to grow over time as more are implemented. diff --git a/x-pack/docs/en/rollup/understanding-groups.asciidoc b/x-pack/docs/en/rollup/understanding-groups.asciidoc index d6eef54fab87e..f57f905ae04c8 100644 --- a/x-pack/docs/en/rollup/understanding-groups.asciidoc +++ b/x-pack/docs/en/rollup/understanding-groups.asciidoc @@ -1,6 +1,8 @@ [[rollup-understanding-groups]] == Understanding Groups +experimental[] + To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force the admin to make decisions about what metrics to rollup and on what interval. E.g. The average of `cpu_time` on an hourly basis. This is limiting; if, at a future date, the admin wishes to see the average of `cpu_time` on an hourly basis _and partitioned by `host_name`_,