Rename filebeat apache2 module to apache #9402

jsoriano · 2018-12-05T15:08:03Z

The modules for Apache HTTP Server are called apache in metricbeat
and apache2 in filebeat. Use the same name for both modules.

Fixes #9395.

ruflin

Before merging this we need to think through on what our expected upgrade path here is. What happens if users upgrade from 6.x to 7.x.

ruflin · 2018-12-05T15:30:46Z

dev-tools/ecs-migration.yml

@@ -142,31 +142,31 @@

 ## Apache

- from: apache2.access.user_name
+- from: apache.access.user_name


So this becomes now interesting. This is for compatibility with 6.x. In 6.x it will stay apache2. Now comes the tricky part: We will have a mix of apache2 and apache fields. What should we do?

Not only the fields, but also configurations referring to both names should work during at least some versions (6.7?).
Shall we duplicate the module during some versions?

@ruflin easier, I have reverted the changes in the ECS migration, and I have added aliases for the rest of fields, so we use the same strategy than for ECS.

For config migration I am still not sure how to do it.

@ruflin I am trying the approach of adding an option to module.yml to indicate that a module has been moved, this keeps old configurations working, see 9af3f75.
If you are ok with this approach I'll open an specific PR just for that.

PR for moved modules #9432

elasticmachine · 2018-12-06T10:50:06Z

Pinging @elastic/infrastructure

This reverts commit 23c3bd4.

If a module needs to be moved, existing configuration will stop working, this change adds an option to mark an old module as moved to other name. This is being considered for the movement of apache2 module to apache to be coherent with metricbeat module (elastic#9402).

jsoriano · 2018-12-07T11:15:25Z

filebeat/fileset/modules.go

@@ -26,6 +26,7 @@ import (
 	"strings"


Changes on this file if done will be done in #9432.

ruflin · 2018-12-07T11:40:14Z

I had a chat with @jsoriano about the migration from 6.x to 7.x and it should be working the same ways as for any other module based on the proposal here.

Migration should work as expected.

ruflin · 2018-12-07T11:41:19Z

PR #8963 should be merged first to reduce the surface of this change.

If a module needs to be moved, existing configuration will stop working, this change adds an option to mark an old module as moved to other name. This is being considered for the movement of apache2 module to apache to be coherent with metricbeat module (#9402).

…name

jsoriano · 2019-01-09T16:55:05Z

@ruflin this would be mostly ready, my main doubt is what to do with dashboards, not sure if I should rename fields there.
How are we going to support deployments with different versions of filebeat? I guess the same will happen with ECS. If filebeat 6.X and 7.X are used in the same deployment, the same dashboard will need to access different fields.
Do dashboards somehow support aliases?

webmat · 2019-01-09T18:44:49Z

@jsoriano Yes, aggregations support aliases, and all visualizations are built on that. The only part that aliases don't help us with, in the Kibana Objects, is the saved searches. The saved search columns are _source field names. Aliases don't apply to _source at all.

ruflin · 2019-01-10T08:10:09Z

What we currently have is that if someone upgrades from 6.x to 7.x, he keeps his old dashboards from 6.x and enables the migration layer. As we have added all alias for the apache2 fields, things should keep working as is.

For dashboards etc. I'm in the progress of writing a migration scripts based on the ecs-migration.yml file. I just realised now that this is missing in this PR and should be added to make sure we rename the fields correctly. Potentially it just needs update to point to the "apache" instead of "apache2" fields.

filebeat/module/apache/_meta/fields.yml

ruflin

LGTM. I suggest we merge this and do a ECS cleanup again afterwards.

@webmat Let us know if you have some objections.

ruflin · 2019-01-11T11:17:19Z

filebeat/module/apache/access/_meta/fields.yml

+  description: >
+    Contains fields for the Apache HTTP Server access logs.
+  fields:
+    - name: remote_ip


@webmat We should follow up on this and convert it to ECS.

Yes, I have a note at the end of the Fb modules list (in #8655) to convert to using .address in all remaining places (it was a beta 2 addition, and came from the first migrations, like this one)

webmat

LGTM, and agree with a round of ECS cleanup afterwards

jsoriano · 2019-01-12T15:00:11Z

Ok, I am merging this, and lets polish when all pieces are in place. Thanks for the feedback and reviews!

jsoriano added module Filebeat Filebeat v7.0.0 labels Dec 5, 2018

jsoriano self-assigned this Dec 5, 2018

jsoriano force-pushed the filebeat-apache2-rename branch 2 times, most recently from 34beeb1 to 4081e91 Compare December 5, 2018 15:25

ruflin previously requested changes Dec 5, 2018

View reviewed changes

jsoriano added the Team:Integrations Label for the Integrations team label Dec 6, 2018

jsoriano added 6 commits December 6, 2018 16:24

Rename filebeat apache2 module to apache

3d7e71a

Add aliases from old fields to new ones

8724749

Use old names in apache dashboard for backwards compatibility

7d98126

Remove reference to apache2 in tests

23c3bd4

Move image used in docs

8637d2f

Revert changes in apache2 paths

be2f162

jsoriano force-pushed the filebeat-apache2-rename branch from a2b0ef4 to be2f162 Compare December 6, 2018 15:32

jsoriano added 3 commits December 6, 2018 17:03

Make aliases for old apache2 fields

436bc15

Revert "Remove reference to apache2 in tests"

6442d5f

This reverts commit 23c3bd4.

Add option for moved filebeat modules

9af3f75

jsoriano added the in progress Pull request is currently in progress. label Dec 6, 2018

jsoriano mentioned this pull request Dec 7, 2018

Add option to support moved filebeat modules #9432

Merged

jsoriano commented Dec 7, 2018

View reviewed changes

filebeat/fileset/modules.go Outdated

@@ -26,6 +26,7 @@ import (

"strings"

Copy link

Member Author

jsoriano Dec 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes on this file if done will be done in #9432.

alvarolobato added the breaking change label Dec 10, 2018

Merge remote-tracking branch 'origin/master' into filebeat-apache2-re…

3b65930

…name

Don't generate fields assets on modules without fields

11e6c07

jsoriano requested review from a team as code owners January 8, 2019 17:42

urso removed the request for review from a team January 8, 2019 22:20

jsoriano added 3 commits January 9, 2019 11:45

Merge remote-tracking branch 'origin/master' into filebeat-apache2-re…

3d2421c

…name

Update aliases for apache error fileset

6248807

Remove aliases not needed in error fileset

6cc43a1

jsoriano force-pushed the filebeat-apache2-rename branch from 8ea885d to 6cc43a1 Compare January 9, 2019 12:03

Remove wrong alias to error client

84a7294

ruflin reviewed Jan 10, 2019

View reviewed changes

filebeat/module/apache/_meta/fields.yml Show resolved Hide resolved

Add migration for moved fields

d7c8bbc

ruflin approved these changes Jan 11, 2019

View reviewed changes

jsoriano mentioned this pull request Jan 11, 2019

Introduce migration script for data in Kibana files #9998

Merged

webmat approved these changes Jan 11, 2019

View reviewed changes

kaiyan-sheng approved these changes Jan 12, 2019

View reviewed changes

jsoriano merged commit 1ce20cb into elastic:master Jan 12, 2019

jsoriano deleted the filebeat-apache2-rename branch January 12, 2019 15:03

karenzone mentioned this pull request Jan 13, 2019

[Doc] Fix broken link in filebeat module elastic/logstash#10286

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rename filebeat apache2 module to apache #9402

Rename filebeat apache2 module to apache #9402

jsoriano commented Dec 5, 2018

ruflin left a comment

ruflin Dec 5, 2018

jsoriano Dec 5, 2018

jsoriano Dec 5, 2018

jsoriano Dec 5, 2018

jsoriano Dec 6, 2018

jsoriano Dec 7, 2018

elasticmachine commented Dec 6, 2018

jsoriano Dec 7, 2018

ruflin commented Dec 7, 2018

ruflin commented Dec 7, 2018

jsoriano commented Jan 9, 2019

webmat commented Jan 9, 2019

ruflin commented Jan 10, 2019

ruflin left a comment

ruflin Jan 11, 2019

webmat Jan 11, 2019 •

edited

Loading

webmat left a comment •

edited

Loading

jsoriano commented Jan 12, 2019

Rename filebeat apache2 module to apache #9402

Rename filebeat apache2 module to apache #9402

Conversation

jsoriano commented Dec 5, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Dec 6, 2018

Choose a reason for hiding this comment

ruflin commented Dec 7, 2018

ruflin commented Dec 7, 2018

jsoriano commented Jan 9, 2019

webmat commented Jan 9, 2019

ruflin commented Jan 10, 2019

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Jan 11, 2019 • edited Loading

Choose a reason for hiding this comment

webmat left a comment • edited Loading

Choose a reason for hiding this comment

jsoriano commented Jan 12, 2019

webmat Jan 11, 2019 •

edited

Loading

webmat left a comment •

edited

Loading