Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Monitoring] Metricbeat migration for logstash, beats and apm #40442

Merged

Conversation

chrisronline
Copy link
Contributor

@chrisronline chrisronline commented Jul 5, 2019

This PR is step 3 of the Metricbeat setup wizard.

It adds MB migration UX for logstash, beats and APM, including net new user experience as well as migration from internal collection.

This PR builds upon the functionality added in the two step 2 PRs, part 1 and part 2.

In addition, this PR also adds a button to the bottom of each stack products' listing page that allows the user to access the setup wizard to add monitoring for a new node/instance:
kibana
logstash
beats
apm
es

Also, this PR takes advantage of the fact that we're attempting to detect if certain stack products exist, based on known indices. This wasn't something that mattered in previous PRs as we can know for certain that a Kibana and ES exist if they are in the stack monitoring UI.

logstash

beats

apm

Lastly, this PR adds code to disable entering setup mode if on cloud.

During testing, we realized that logstash and beats can potentially contain mb monitoring documents without a cluster_uuid (which would put those instances in the standalone cluster) so we built a flow in the wizard to let the user know the location of this instance/node will change. In this flow, the user is required to check the checkbox to finish the wizard. Here is what that looks like:
Screen Shot 2019-07-26 at 11 30 48 AM
Screen Shot 2019-07-26 at 11 30 42 AM

Testing

The first thing to talk about is enabling this "Setup Mode". The most straight forward way is to just update the query string in the url, by adding this to the global state param: ,inSetupMode:!t. A final, working URL will look something like: http://localhost:5601/app/monitoring#/elasticsearch/nodes?_g=(cluster_uuid:qtoNYcCxQdK-9BGIdPN_lw,inSetupMode:!t)

  • Please test all three newly supported stack products: Logstash, Beats, and APM
  • For each stack product, please test both flows: net new user (no existing monitoring data for the stack product) and migration (existing, internally collected monitoring data)
  • For net new user flow, there are two potential sub-flows
    • We detect the stack product might exist and provide messaging to that effect
    • We are unable to detect any product and the only option is the add monitoring button at the bottom of the listing page
  • For APM, follow these steps:
    1. Clone and run https://github.com/elastic/apm-server following instructions in README
    2. Install the apm client within Kibana itself
      1. yarn add elastic-apm-node
      2. Add the following code to the top of the filescripts/kibana.js: require('elastic-apm-node').start({})

Existing Issues

  • I ran into an issue with Beats in the migration flow, specifically in the disable internal collection step. Beats is designed to only make the connection to the ES output (assuming it has an ES output configured) once data is ready to be sent. It's possible that in the process of restarting the beat (after disabling internal collection) will not yield a new event, so the ES output connection is never registered which causes issues in this flow (as MB will still collect from the beats API but the beat API will not report a valid cluster_uuid as a result). This can be addressed in testing by deleting the data directory in the beat and restarting.

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@spalger
Copy link
Contributor

spalger commented Jul 19, 2019

retest

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@chrisronline chrisronline force-pushed the monitoring/mb_setup_wizard_step3 branch from 1be61d7 to bc03b08 Compare July 22, 2019 17:00
@elasticmachine
Copy link
Contributor

💔 Build Failed

@chrisronline chrisronline requested a review from igoristic July 22, 2019 18:05
@chrisronline chrisronline self-assigned this Jul 22, 2019
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

<EuiText>
<FormattedMessage
id="xpack.monitoring.metricbeatMigration.apmInstructions.metricbeatSecuritySetup"
defaultMessage="If security features are enabled, there may be more setup required.{link}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you might need a space after the dot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a space in the link fragment actually that should get the job done, unless you're not seeing a space when running it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was just looking from the code point of view, but I understand now. Will test in a little bit 👍


const enableMetricbeatModuleStep = {
title: i18n.translate('xpack.monitoring.metricbeatMigration.apmInstructions.enableMetricbeatModuleTitle', {
defaultMessage: 'Enable and configure the beat x-pack module in Metricbeat'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe capitalize Beat?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure in this case as the module is called beat-xpack. I can go either way I guess. Anyone else have thoughts here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, Beat is still a proper name in the same way that we capitalize Metricbeat. Though, I will grant you that the naming scheme that was chosen makes this feel a little awkward. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed!

<EuiText>
<FormattedMessage
id="xpack.monitoring.metricbeatMigration.beatsInstructions.metricbeatSecuritySetup"
defaultMessage="If security features are enabled, there may be more setup required.{link}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might need space after the dot

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my note from the other comment about needing a space after the dot

<p>
<FormattedMessage
id="xpack.monitoring.metricbeatMigration.logstashInstructions.disableInternalCollection.note"
defaultMessage="You'll need to restart logstash after making this change"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe capital Logstash?

<EuiText>
<FormattedMessage
id="xpack.monitoring.metricbeatMigration.logstashInstructions.metricbeatSecuritySetup"
defaultMessage="If security features are enabled, there may be more setup required.{link}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Space after dot?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my other comment about this

@igoristic
Copy link
Contributor

Everything looks good! Great job 💪

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@chrisronline chrisronline merged commit 33fea2f into elastic:master Jul 31, 2019
@chrisronline chrisronline deleted the monitoring/mb_setup_wizard_step3 branch July 31, 2019 18:17
chrisronline added a commit to chrisronline/kibana that referenced this pull request Jul 31, 2019
chrisronline added a commit to chrisronline/kibana that referenced this pull request Aug 1, 2019
…c#40442)

* Support for logstash

* Beats support

* Fix cherry-pick api issue

* Support for logstash

* Updates for beats and logstash

* APM migration working

* Tweaks for beats migration

* Update copy for setup new button

* If on cloud, disable setup mode

* Handle new beat flow better

* Better phrasing for APM

* Add beat type to disable step

* Fix i18n issue

* Fix jest tests

* Fix api tests

* PR feedback

* Update copy

* Remove unnecessary code

* Undo changes that are now in a separate PR

* Disable more links

* Fix overview link for logstash

* PR feedback

* Fix tests

* PR feedback

* PR feedback

* Capitalize Beat per PR feedback
chrisronline added a commit that referenced this pull request Aug 1, 2019
#42455)

* Support for logstash

* Beats support

* Fix cherry-pick api issue

* Support for logstash

* Updates for beats and logstash

* APM migration working

* Tweaks for beats migration

* Update copy for setup new button

* If on cloud, disable setup mode

* Handle new beat flow better

* Better phrasing for APM

* Add beat type to disable step

* Fix i18n issue

* Fix jest tests

* Fix api tests

* PR feedback

* Update copy

* Remove unnecessary code

* Undo changes that are now in a separate PR

* Disable more links

* Fix overview link for logstash

* PR feedback

* Fix tests

* PR feedback

* PR feedback

* Capitalize Beat per PR feedback
@chrisronline
Copy link
Contributor Author

Backport:

7.x: 1597b30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes review Team:Monitoring Stack Monitoring team v7.4.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants