Introduce data stream options and failure store configuration classes

[gmarouli]

In order to facilitate enabling and disabling the failure store & component template composition, we introduce new metadata classes that can support a more extensible failure store configuration.

We would like to introduce data stream options. Data stream options capture the configuration of data stream level (smaller and larger) features, such the failure store and in the future data stream lifecycle. They are different than settings because they are applied on a data stream level and not per backing index.

This PR is only setting the basic classes to enable follow up PRs that will actually use them.

Examples, these are not final, they are only used to help visualise a potential direction:

GET _data_stream/my-*/_options
{
  "data_streams": [
    {
      "name": "my-non-opinionated-ds",
      "options": {
      }
    },
    {
      "name": "my-fs",
      "options": {
        "failure_store": {
          "enabled": true
        }
      }
    },
    {
      "name": "my-no-fs",
      "options": {
        "failure_store": {
          "enabled": false
        }
      }
    }
  ]
}

// If we decide to add lifecycle here too:
PUT _data_stream/my-fs/_options
{
   "failure_store": {
     "enabled": true
   },
   "lifecycle": {
   }
}

What we see above are 3 data streams:

• my-fs with the failure store explicitly enabled
• my-no-fs with the failure store explicitly disabled, and
• my-non-opinionated-ds which does not specify what to do with the failure store, so for now it means failure store disabled but that could change in the future.

Template composition examples pending

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

I left one minor typo suggestion, other than that it LGTM :).
I do think it makes sense to wait with merging this, because the DataStreamOptions implementation might differ a bit depending on the ongoing discussions we're having. What do you think?

[dakrone]

I left a couple of comments. If we decide we want to include lifecycle options (which I would like to, but I know it's up for discussion), then it'd be interesting to see that here, even if it's not used yet.

[gmarouli]

I left one minor typo suggestion, other than that it LGTM :). I do think it makes sense to wait with merging this, because the DataStreamOptions implementation might differ a bit depending on the ongoing discussions we're having. What do you think?

Yes, definitely, I am open to change it. This is supposed to help identify red flags by having something more concrete. With this I can at least keep preparing for the API even if a few things could change, without feeling I am going on a wrong direction.

[gmarouli]

I left a couple of comments. If we decide we want to include lifecycle options (which I would like to, but I know it's up for discussion), then it'd be interesting to see that here, even if it's not used yet.

Happy to include it here just for demonstration purposes. I can always revert the commit if we think it doesn't fit anymore.

[gmarouli]

@elasticmachine update branch

[gmarouli]

@elasticmachine update branch

[gmarouli]

@elasticmachine update branch

[gmarouli]

@dakrone & @jbaiera could you give this another look please. Thank you ☺️ .

[gmarouli]

@elasticmachine update branch

[gmarouli]

@elasticmachine update branch

[gmarouli]

@elasticmachine update branch

[dakrone]

LGTM, I left one comment

[dakrone]

Is there a reason you went with a static method rather than a constructor? We've been moving to constructors for this more and more in the code.

[gmarouli]

Oh I wasn't aware we prefer constructors. I preferred the static method because it allows more flexibility, so I thought it was better to always use a static method to facilitate all cases.

However, if this is not the direction we are following and in this case there is no blocker in using a constructor, I will change it to use a constructor.

[nielsbauman]

I'm personally also more of a fan of static methods (because in some cases we need to use a static method rather than a constructor).

[gmarouli]

@elasticmachine update branch

[gmarouli]

@jbaiera if you want to see how it looks with the lifecycle as well, check the diff in a26bb6d (the revert commit)

[jbaiera]

One of the things we're talking about doing is adding a cluster setting that will contain index patterns which will state that failure store is enabled for any matching indices by default. The plan is to fall back to using that cluster setting in the case that failure stores haven't been explicitly enabled or disabled via templates at creation time, or in the state.

To save on headache going forward, should this boolean be an optional Boolean?

[gmarouli]

Good point! thinking further about this, I would expect that the following to be the not explicitly set:

• Explicitly enabled:

PUT _data_stream/my-fs/_options
{
   "failure_store": {
     "enabled": true
   }
}

• Explicitly disabled:

PUT _data_stream/my-fs/_options
{
   "failure_store": {
     "enabled": false
   }
}

• No explicit configuration:

PUT _data_stream/my-fs/_options
{}

This follows the way that the lifecycle works. In template composition, a user can set failure_store: null to remove any configuration set by previous templates.

Since this is a new configuration type we could always decide we would like to change the way we work with this, but I thought we should at least consider it.

What do you think?

[jbaiera]

Ah I see, since there is only one property we can just assume the entire section will be removed from options if it's not present. I'm good with that

[gmarouli]

@elasticmachine update branch

[jbaiera]

LGTM!

[jbaiera]

Ah I see, since there is only one property we can just assume the entire section will be removed from options if it's not present. I'm good with that

[gmarouli]

@elasticmachine update branch