Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate Kibana index pattern #2122

Merged
merged 1 commit into from
Aug 4, 2016
Merged

Generate Kibana index pattern #2122

merged 1 commit into from
Aug 4, 2016

Conversation

monicasarbu
Copy link
Contributor

@monicasarbu monicasarbu commented Jul 28, 2016 •
edited
Loading

This PR creates a python script to generate the index pattern in json format from the fields.yml. The generated index pattern is needed by Kibana to know how to format the fields, think of a percentage value.

As Kibana supports number, string and boolean types, all the variations of half_float, float, integer, long become number in the index-pattern.

For special fields that require a special format (like percent, bytes), usually you need to define the format manually for each field in Kibana in order for Kibana to know how to display a number when it's a percentage (using %) or in MB/GB for bytes values. To automate the process, this PR lets you define the format of the field in the format option.

- name: used.bytes
   type: long
   format: bytes
   description: >
        Used memory.

- name: used.pct
   type: half_float
   format: percent
   description: >
        The percentage of used memory.

This PR is another way to solve elastic/beats-dashboards#117

@monicasarbu monicasarbu added the in progress Pull request is currently in progress. label Jul 28, 2016
@monicasarbu monicasarbu force-pushed the generate_index_pattern branch from 6ed8c43 to fda669d Compare July 28, 2016 17:46
@monicasarbu monicasarbu changed the title Generate index pattern Generate Kibana index pattern Jul 28, 2016
@monicasarbu monicasarbu force-pushed the generate_index_pattern branch from eed7b93 to 3e2b249 Compare July 28, 2016 17:52
@monicasarbu monicasarbu added review in progress Pull request is currently in progress. and removed in progress Pull request is currently in progress. labels Jul 28, 2016
@monicasarbu monicasarbu force-pushed the generate_index_pattern branch from 3723183 to 7aba4f4 Compare July 28, 2016 18:17
@monicasarbu monicasarbu removed the in progress Pull request is currently in progress. label Jul 28, 2016
tsg
tsg reviewed Jul 29, 2016
View reviewed changes
libbeat/scripts/generate_index_pattern.py
}
if "type" in desc:
if desc["type"] in ["half_float", "float", "integer", "long"]:
field["type"] = "integer"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "integer" correct here? Shouldn't it be "number"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good spot, @tsg. Thanks!

@tsg
Copy link
Contributor

tsg commented Jul 29, 2016

LGTM, great work! I see only the Packetbeat index was re-generated, should we do them all in this PR?

Also, needs a rebase.

@monicasarbu
Copy link
Contributor Author

Yes, I will do for all the Beats in the same PR.

@monicasarbu monicasarbu added the in progress Pull request is currently in progress. label Jul 29, 2016
@monicasarbu monicasarbu force-pushed the generate_index_pattern branch 2 times, most recently from b865ce1 to 158c66f Compare July 29, 2016 11:02
@monicasarbu
Copy link
Contributor Author

With this change the import-dashboards.sh doesn't work anymore to load the dashboards and it needs to be changed in a different PR.

@monicasarbu monicasarbu added in progress Pull request is currently in progress. and removed in progress Pull request is currently in progress. labels Jul 29, 2016
ruflin
ruflin reviewed Aug 2, 2016
View reviewed changes
metricbeat/docs/fields.asciidoc
@@ -1901,6 +1911,8 @@ CPU Core number.

type: half_float

format: percent
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's nice to have this also in the docs 👍

@ruflin
Copy link
Member

ruflin commented Aug 2, 2016

LGTM. Great work. Could you rebase?

@monicasarbu monicasarbu removed the in progress Pull request is currently in progress. label Aug 3, 2016
@monicasarbu monicasarbu force-pushed the generate_index_pattern branch from 158c66f to 55b5004 Compare August 3, 2016 12:10
@monicasarbu
Copy link
Contributor Author

Rebased.

ruflin
ruflin reviewed Aug 3, 2016
View reviewed changes
CHANGELOG.asciidoc
@@ -45,6 +45,17 @@ https://github.com/elastic/beats/compare/v5.0.0-alpha5...master[Check the HEAD d

*Affecting all Beats*

- Periodically log internal metrics. {pull}1955[1955]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are these changelog entries coming from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it looks like "git rebase master" didn't do a good job. They were part of alpha5. Thanks for noticing it. I am fixing it manually.

@monicasarbu
Generate Kibana index pattern from etc/kibana/fields.yml
c9cde58 
The format of each field can be specified in fields.yml using the `format` option.

Generate Kibana index pattern for all the elastic beats by running `make update`
@monicasarbu monicasarbu force-pushed the generate_index_pattern branch from 4761ee4 to c9cde58 Compare August 3, 2016 14:07
@tsg tsg merged commit 32446ed into elastic:master Aug 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@monicasarbu @tsg @ruflin