Adding data_lifecycle to the _xpack/usage API #96177

masseyke · 2023-05-16T17:52:53Z

This adds a data_lifecycle section to the _xpack/usage API, giving basic information about data lifecycles in the cluster. The data looks something like:

    "data_lifecycle": {
        "available": true,
        "enabled": true,
        "count": 1,
        "default_rollover_used": true,
        "retention": {
           "minimum_millis": 360000,
           "maximum_millis": 360000,
           "average_millis": 360000.0
        }
    },

elasticsearchmachine · 2023-05-16T17:53:44Z

Hi @masseyke, I've created a changelog YAML for you.

elasticsearchmachine · 2023-05-17T14:15:51Z

Pinging @elastic/es-data-management (Team:Data Management)

gmarouli

Thank you for working on this @masseyke ! I am so glad we will have data so early in the process. I have some comments in the PR, I am curious what you think :)

Also, I was wondering if we could still write a test like x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/health/10_usage.yml using templates and data stream creation without needing the APIs that might not be available. What do you think?

gmarouli · 2023-05-18T09:40:39Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java

@@ -545,6 +546,7 @@ public List<NamedWriteableRegistry.Entry> getNamedWriteables() {
            ),
            // Data Streams
            new NamedWriteableRegistry.Entry(XPackFeatureSet.Usage.class, XPackField.DATA_STREAMS, DataStreamFeatureSetUsage::new),
+            new NamedWriteableRegistry.Entry(XPackFeatureSet.Usage.class, XPackField.DATA_LIFECYCLE, DataLifecycleFeatureSetUsage::new),


We should only register this if the feature flag is on right?

I wasn't sure -- I always loaded it because the API has an enabled field. But I guess that enabled field is not actually meant for feature flags, is it? I'll hide all of these if the feature flag is off.

I think so too.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java

...ore/src/main/java/org/elasticsearch/xpack/core/action/DataLifecycleUsageTransportAction.java

gmarouli · 2023-05-18T09:54:07Z

...k/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/XPackUsageFeatureAction.java

@@ -57,6 +58,7 @@ public class XPackUsageFeatureAction extends ActionType<XPackUsageFeatureRespons
        ANALYTICS,
        CCR,
        DATA_STREAMS,
+        DATA_LIFECYCLE,


Behind feature flag?

gmarouli · 2023-05-18T10:21:30Z

x-pack/qa/usage/src/javaRestTest/java/org/elasticsearch/xpack/usage/XPackUsageIT.java

+ * This class is meant to test the _xpack/usage API. That API pulls in code from various plugins so it cannot be integration tested
+ * within x-pack/plugin/core.
+ */
+public class XPackUsageIT extends ESRestTestCase {


Since this is testing only DLM should we make the name a bit more specific?

I named the test method testDataLifecycle. But I named the test XPackUsageIT so that people could add other xpack usage tests here over time. I had hoped to find an XPackUsageIT that I could just add my method to, so I figured someone else might in the future.

On that note though, this is a very expensive test -- it can take about 30 seconds -- and I'm not sure it's actually worth the cost. We have similar (not quite as realistic) coverage in DataLifecycleUsageTransportActionIT (~10 seconds), and serialization coverage in DataLifecycleFeatureSetUsageTests (fast). I'm not sure this test is worth the cost for such simple code with coverage elsewhere.

masseyke · 2023-05-18T13:15:44Z

Also, I was wondering if we could still write a test like x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/health/10_usage.yml using templates and data stream creation without needing the APIs that might not be available. What do you think?

I had originally intended to do that, but I wanted to add / remove multiple datastreams, and worried that it would turn into a huge mess in yaml. So I added XPackUsageIT::testDataLifecycle to do the same thing, but with functions (so it didn't get too big to follow). I'm not all that familiar with the yaml tests though -- is there an easy way to write the equivalent of functions there? (and then see my note on my concerns over whether this test even justifies its cost -- I assume the yaml test would have similar cost)

gmarouli · 2023-05-24T13:49:22Z

Also, I was wondering if we could still write a test like x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/health/10_usage.yml using templates and data stream creation without needing the APIs that might not be available. What do you think?

I had originally intended to do that, but I wanted to add / remove multiple datastreams, and worried that it would turn into a huge mess in yaml. So I added XPackUsageIT::testDataLifecycle to do the same thing, but with functions (so it didn't get too big to follow). I'm not all that familiar with the yaml tests though -- is there an easy way to write the equivalent of functions there? (and then see my note on my concerns over whether this test even justifies its cost -- I assume the yaml test would have similar cost)

I see your concern, I was hoping that with the yaml test we could use the existing spinned up clusters and then just add a few data streams and see we get the correct metrics. Would that be so expensive?

It's more of a sanity test not a really detailed one.

PS: apologies for the late response.

masseyke · 2023-05-25T22:12:53Z

I swapped out XPackUsageIT for a yaml rest test.

gmarouli

LGTM, if I am not mistaken, the transport version in the serialisable object needs to be updated. But it looks good otherwise.

And a small comment about documentation.

...ore/src/main/java/org/elasticsearch/xpack/core/datastreams/DataLifecycleFeatureSetUsage.java

x-pack/qa/usage/src/main/java/org/elasticsearch/xpack/usage/UsageQA.java

Co-authored-by: Mary Gouseti <[email protected]>

….com:masseyke/elasticsearch into feature/adding-data-lifecycle-to-xpack-usage

gmarouli

LGTM! Thank you for working on this @masseyke!

gmarouli · 2023-05-30T10:47:11Z

@masseyke I took the liberty to merge your PR.

Adding data_lifecycle to the _xpack/usage API

49cab23

masseyke marked this pull request as draft May 16, 2023 17:53

elasticsearchmachine added the v8.9.0 label May 16, 2023

masseyke added >enhancement :Data Management/DLM labels May 16, 2023

masseyke added 5 commits May 16, 2023 12:53

Update docs/changelog/96177.yaml

9d70735

fixing license

7b9c4bf

adding an integration test for DataLifecycleUsageTransportAction

e040fba

cleanup

786cf21

spotlessApply

56ca4a1

masseyke marked this pull request as ready for review May 17, 2023 14:15

masseyke requested a review from gmarouli May 17, 2023 14:15

elasticsearchmachine added the Team:Data Management Meta label for data/management team label May 17, 2023

gmarouli reviewed May 18, 2023

View reviewed changes

masseyke added 3 commits May 18, 2023 14:32

putting DLM telemetry behind the feature flag

8a3a421

merging main

e50e6e4

exposing default_rollover_used rather than rollover_config

9654851

masseyke requested a review from gmarouli May 18, 2023 20:44

masseyke added 2 commits May 18, 2023 16:25

exposing default_rollover_used rather than rollover_config

fd15459

spotlessApply

2350562

masseyke added 2 commits May 25, 2023 15:36

merging main

aa8677e

replacing XPackUsageIT with a yaml rest test

93a33ee

gmarouli requested changes May 26, 2023

View reviewed changes

masseyke added 2 commits May 26, 2023 08:50

removing now-unused files

2a22ead

Merge branch 'main' into feature/adding-data-lifecycle-to-xpack-usage

3921398

masseyke and others added 2 commits May 26, 2023 08:52

Updating TransportVersion

e3f4bc2

Co-authored-by: Mary Gouseti <[email protected]>

Merge branch 'feature/adding-data-lifecycle-to-xpack-usage' of github…

0cf4609

….com:masseyke/elasticsearch into feature/adding-data-lifecycle-to-xpack-usage

masseyke requested a review from gmarouli May 26, 2023 14:36

gmarouli approved these changes May 29, 2023

View reviewed changes

gmarouli merged commit 17ff9f0 into elastic:main May 30, 2023

masseyke deleted the feature/adding-data-lifecycle-to-xpack-usage branch June 5, 2023 13:59

gmarouli mentioned this pull request Jun 6, 2023

[Meta] Data Lifecycle Management #93596

Closed

19 tasks

masseyke mentioned this pull request Jun 9, 2023

[CI] UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/90_ml_data_frame_analytics_crud/Get old regression job} failing #96520

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding data_lifecycle to the _xpack/usage API #96177

Adding data_lifecycle to the _xpack/usage API #96177

masseyke commented May 16, 2023 •

edited by gmarouli

Loading

elasticsearchmachine commented May 16, 2023

elasticsearchmachine commented May 17, 2023

gmarouli left a comment

gmarouli May 18, 2023

masseyke May 18, 2023

gmarouli May 24, 2023

gmarouli May 18, 2023

gmarouli May 18, 2023

masseyke May 18, 2023

masseyke commented May 18, 2023

gmarouli commented May 24, 2023

masseyke commented May 25, 2023

gmarouli left a comment

gmarouli left a comment

gmarouli commented May 30, 2023

Adding data_lifecycle to the _xpack/usage API #96177

Adding data_lifecycle to the _xpack/usage API #96177

Conversation

masseyke commented May 16, 2023 • edited by gmarouli Loading

elasticsearchmachine commented May 16, 2023

elasticsearchmachine commented May 17, 2023

gmarouli left a comment

Choose a reason for hiding this comment

gmarouli May 18, 2023

Choose a reason for hiding this comment

masseyke May 18, 2023

Choose a reason for hiding this comment

gmarouli May 24, 2023

Choose a reason for hiding this comment

gmarouli May 18, 2023

Choose a reason for hiding this comment

gmarouli May 18, 2023

Choose a reason for hiding this comment

masseyke May 18, 2023

Choose a reason for hiding this comment

masseyke commented May 18, 2023

gmarouli commented May 24, 2023

masseyke commented May 25, 2023

gmarouli left a comment

Choose a reason for hiding this comment

gmarouli left a comment

Choose a reason for hiding this comment

gmarouli commented May 30, 2023

masseyke commented May 16, 2023 •

edited by gmarouli

Loading