Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] ILM with configuration per index #9616

Closed
wants to merge 2 commits into from

Conversation

simitt
Copy link
Contributor

@simitt simitt commented Dec 18, 2018

This PR intends to:

  • refactor template export to reuse template loader
  • move ilm logic to its own package
  • change ilm logic to load write alias for every index that has a rollover_policy defined. Add more default policies so they are also usable for APM. ILM changes have been discussed with @ruflin upfront.

TODO: add more tests

cc @elastic/apm-server

Example config metricbeat:

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "metricbeat-%{[beat.version]}"
  rollover_policy: "default"

Example config apm:

output.elasticsearch:
  indices:
  - index: "apm-%{[beat.version]}-sourcemap"
    when.contains:
      processor.event: "sourcemap"

  - index: "apm-%{[beat.version]}-error"
    when.contains:
      processor.event: "error"
    rollover_policy: "deleteAfter10Days"

  - index: "apm-%{[beat.version]}-transaction"
    when.contains:
      processor.event: "transaction"
    rollover_policy: "deleteAfter1Year"

  - index: "apm-%{[beat.version]}-span"
    when.contains:
      processor.event: "span"
    rollover_policy: "deleteAfter10Days"

  - index: "apm-%{[beat.version]}-metric"
    when.contains:
      processor.event: "metric"
    rollover_policy: "deleteAfter1Year"

  - index: "apm-%{[beat.version]}-onboarding}"
    when.contains:
      processor.event: "onboarding"

Enable ILM for all indices that have a `rollover_policy` defined, and a
valid index name.
Add more default rollover policies.
@simitt simitt requested review from ph, urso and ruflin December 18, 2018 11:08
@simitt simitt requested a review from a team as a code owner December 18, 2018 11:08
return nil
}

func (l *Loader) LoadWriteAlias() error {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Loader.LoadWriteAlias should have comment or be unexported

}, nil
}

func (l *Loader) LoadPolicies() error {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Loader.LoadPolicies should have comment or be unexported

GetVersion() common.Version
}

type Loader struct {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type Loader should have comment or be unexported

"github.com/elastic/beats/libbeat/template"
)

type ESClient interface {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported type ESClient should have comment or be unexported

@ruflin ruflin added in progress Pull request is currently in progress. libbeat labels Dec 20, 2018
@simitt simitt mentioned this pull request Jan 7, 2019
5 tasks
@simitt
Copy link
Contributor Author

simitt commented Jan 10, 2019

Closing this in favour of #10002

@simitt simitt closed this Jan 10, 2019
@simitt simitt deleted the ilm-with-indices branch May 6, 2019 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress Pull request is currently in progress. libbeat
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants