[Metricbeat] Create Kibana dashboards for ActiveMQ module #14655

mtojek · 2019-11-20T16:14:31Z

Marking as WIP to let you review changes before I dive into testing and dashboards.

Issue: #14510

This PR introduces dashboards for recently developed ActiveMQ (metricsets: broker, topic, queue)

Done:

broker
topic
queue

mtojek · 2019-11-20T16:20:22Z

I wonder if you prefer to see some screenshots here. If so, I can upload them to a side repo.

mtojek · 2019-11-21T08:03:00Z

Screenshots

Broker Overview:
https://raw.githubusercontent.com/mtojek/gh-files/master/elastic/beats/pull-requests/14655/screencapture-localhost-5601-app-kibana-2019-11-21-08_54_17.png

Queues Overview:
https://raw.githubusercontent.com/mtojek/gh-files/master/elastic/beats/pull-requests/14655/screencapture-localhost-5601-app-kibana-2019-11-21-08_55_34.png

Topics Overview:
https://raw.githubusercontent.com/mtojek/gh-files/master/elastic/beats/pull-requests/14655/screencapture-localhost-5601-app-kibana-2019-11-21-08_55_02.png

exekias · 2019-11-21T09:57:43Z

ey @mtojek we normally upload the screenshots of the dashboards and put them in the docs. See this as an example: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-sqs.html

It should also help with the review

mtojek · 2019-11-21T14:08:54Z

Thanks @exekias for the suggestion. I've added some screenshots.

exekias · 2019-11-22T09:30:07Z

Thank you! they are looking great :) Could you please add units for enqueue time visualizations? You can access them under the Options tab:

Also I saw that Broker memory usage is 0, is that correct? Adding units there would be helpful too

mtojek · 2019-11-22T09:47:52Z

I will add missing units in a moment. Also, verified the Broker memory usage. I've sent ~10000 messages to raise MemoryPercentUsage to 1%:

{"request":{"mbean":"org.apache.activemq:brokerName=*,type=Broker","type":"read"},"value":{"org.apache.activemq:brokerName=localhost,type=Broker":{"StatisticsEnabled":true,"TemporaryQueueSubscribers":[],"TotalConnectionsCount":3,"TotalMessageCount":10153,"TempPercentUsage":0,"MemoryPercentUsage":1,"TransportConnectors":{"openwire":"tcp:\/\/dfb25e965e8c:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600","amqp":"amqp:\/\/dfb25e965e8c:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600","mqtt":"mqtt:\/\/dfb25e965e8c:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600","stomp":"stomp:\/\/dfb25e965e8c:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600","ws":"ws:\/\/dfb25e965e8c:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"},"InactiveDurableTopicSubscribers":[],"StoreLimit":45676818504,"TotalProducerCount":0,"TopicProducers":[],"CurrentConnectionsCount":0,"QueueProducers":[],"JMSJobScheduler":null,"VMURL":"vm:\/\/localhost","UptimeMillis":407715,"TemporaryTopicProducers":[],"TemporaryQueueProducers":[],"TotalDequeueCount":0,"Topics":[{"objectName":"org.apache.activemq:brokerName=localhost,destinationName=ActiveMQ.Advisory.MasterBroker,destinationType=Topic,type=Broker"},{"objectName":"org.apache.activemq:brokerName=localhost,destinationName=ActiveMQ.Advisory.Connection,destinationType=Topic,type=Broker"},{"objectName":"org.apache.activemq:brokerName=localhost,destinationName=ActiveMQ.Advisory.Queue,destinationType=Topic,type=Broker"}],"DurableTopicSubscribers":[],"JobSchedulerStorePercentUsage":0,"Uptime":"6 minutes","QueueSubscribers":[],"BrokerId":"ID:dfb25e965e8c-39859-1574415491731-0:1","AverageMessageSize":1067,"DataDirectory":"\/opt\/apache-activemq-5.15.9\/data","Persistent":true,"TopicSubscribers":[],"TemporaryQueues":[],"BrokerVersion":"5.15.9","BrokerName":"localhost","MinMessageSize":1024,"MemoryLimit":668309914,"Slave":false,"DynamicDestinationProducers":[],"TotalEnqueueCount":10161,"TempLimit":45676806144,"TemporaryTopicSubscribers":[],"TemporaryTopics":[],"JobSchedulerStoreLimit":0,"Queues":[{"objectName":"org.apache.activemq:brokerName=localhost,destinationName=incoming,destinationType=Queue,type=Broker"}],"TotalConsumerCount":0,"StorePercentUsage":0,"MaxMessageSize":1068}},"timestamp":1574415898,"status":200}

I just didn't send enough messages while capturing screenshots.

mtojek · 2019-11-22T15:41:59Z

I've updated dashboards according to changes proposed in #14711 . Once it's merged, I will rebase this PR.

exekias · 2019-11-25T10:59:38Z

metricbeat/docs/modules/activemq.asciidoc

+            if (topic_memory_broker_pct != null) {
+              event.Put("activemq.topic.memory.broker.pct", topic_memory_broker_pct / 100.0)
+            }
+          }


this is great! pinging @jsoriano we should probably support this in the metricset manifest, WDYT?

Yes, I have added an issue for that ~~#14730~~ #14740

I wouldn't add this to the user configuration because we cannot update these files when they are in use.

@mtojek @exekias I have added an issue so we don't forget to move the processors to manifest.yml when possible: #14742

I wouldn't release any module as GA with processors in the configuration.

I don't think #14730 is the right link there 😬 @jsoriano

@kaiyan-sheng thanks! link fixed

) * Kibana dashboard: ActiveMQ broker overview * Fix: mage fmt update * Fix: invalid dashboard JSON * Kibana dashboard: ActiveMQ queues and topics overview * Add dashboard screenshots * Enable time units, pct in dashboards

* [Metricbeat] Create ActiveMQ module (#14580) * Generate bootstrap for broker, queue, topic * Update generated files * Define metricsets in disabler * Prepare docker image for ActiveMQ * Define jolokia endpoint in disabler * Convert to lightweight module * Remove downloaded installation bundle * Fill sample data.json * Refactor metricset fields * Update docs * Fix: do not expose ActiveMQ port * Fix: mage fmt update * Dockerfile: define healthcheck * Move module contents to x-pack * Fix: move docker-compose definition to x-pack * Add basic system test * Fix: update docs * System tests: topic metrics collected * Fix: mage fmt update * System tests: queue, topic metrics collected * Update docs * Fix: use compose_host * Assert that fields are documented * Fix: typo in metricbeat/docs/modules/activemq.asciidoc Co-Authored-By: kaiyan-sheng <[email protected]> * Add CHANGELOG.next entry * Fix: mage fmt update * Fix: import stomp in the function scope (#14703) * Fix: use metrics_path instead of path in system tests (#14690) * [Metricbeat] Create Kibana dashboards for ActiveMQ module (#14655) * Kibana dashboard: ActiveMQ broker overview * Fix: mage fmt update * Fix: invalid dashboard JSON * Kibana dashboard: ActiveMQ queues and topics overview * Add dashboard screenshots * Enable time units, pct in dashboards * Adjust PR reference in CHANGELOG.next * Fix: mage fmt update

mtojek requested a review from a team as a code owner November 20, 2019 16:14

mtojek self-assigned this Nov 20, 2019

mtojek added Team:Integrations Label for the Integrations team in progress Pull request is currently in progress. enhancement module labels Nov 20, 2019

mtojek force-pushed the 14510-create-dashboard branch from 780a2ce to 031c36f Compare November 20, 2019 19:33

mtojek changed the title ~~WIP: [Metricbeat] Create Kibana dashboards for ActiveMQ module~~ [Metricbeat] Create Kibana dashboards for ActiveMQ module Nov 21, 2019

mtojek added review [zube]: In Review and removed in progress Pull request is currently in progress. labels Nov 21, 2019

mtojek force-pushed the 14510-create-dashboard branch from 1f53a51 to dc32f13 Compare November 22, 2019 15:38

mtojek force-pushed the 14510-create-dashboard branch 2 times, most recently from ab45bf9 to fe1cd9a Compare November 24, 2019 18:36

Marcin Tojek added 6 commits November 25, 2019 09:33

Kibana dashboard: ActiveMQ broker overview

9f2f1fd

Fix: mage fmt update

1865115

Fix: invalid dashboard JSON

fa2d6bb

Kibana dashboard: ActiveMQ queues and topics overview

1c13492

Add dashboard screenshots

8eab210

Enable time units, pct in dashboards

bf3d04f

mtojek force-pushed the 14510-create-dashboard branch from fe1cd9a to bf3d04f Compare November 25, 2019 08:34

exekias approved these changes Nov 25, 2019

View reviewed changes

mtojek merged commit 31e3ed4 into elastic:master Nov 25, 2019

zube bot added [zube]: Done and removed [zube]: In Review labels Nov 25, 2019

jsoriano added test-plan Add this PR to be manual test plan v7.6.0 labels Nov 25, 2019

jsoriano mentioned this pull request Nov 25, 2019

Move scripts in light module configurations to manifest.yml #14742

Closed

andresrc removed the [zube]: Done label Nov 25, 2019

andresrc added the needs testing notes label Jan 8, 2020

andresrc unassigned mtojek Jan 14, 2020

ChrsMark self-assigned this Jan 16, 2020

ChrsMark mentioned this pull request Jan 17, 2020

[Metricbeat] ActiveMQ module #14510

Closed

14 tasks

ChrsMark added the test-plan-ok This PR passed manual testing label Jan 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Create Kibana dashboards for ActiveMQ module #14655

[Metricbeat] Create Kibana dashboards for ActiveMQ module #14655

mtojek commented Nov 20, 2019 •

edited

Loading

mtojek commented Nov 20, 2019

mtojek commented Nov 21, 2019

exekias commented Nov 21, 2019

mtojek commented Nov 21, 2019 •

edited

Loading

exekias commented Nov 22, 2019

mtojek commented Nov 22, 2019 •

edited

Loading

mtojek commented Nov 22, 2019

exekias Nov 25, 2019

jsoriano Nov 25, 2019 •

edited

Loading

jsoriano Nov 25, 2019

kaiyan-sheng Nov 25, 2019

jsoriano Nov 25, 2019

[Metricbeat] Create Kibana dashboards for ActiveMQ module #14655

[Metricbeat] Create Kibana dashboards for ActiveMQ module #14655

Conversation

mtojek commented Nov 20, 2019 • edited Loading

mtojek commented Nov 20, 2019

mtojek commented Nov 21, 2019

exekias commented Nov 21, 2019

mtojek commented Nov 21, 2019 • edited Loading

exekias commented Nov 22, 2019

mtojek commented Nov 22, 2019 • edited Loading

mtojek commented Nov 22, 2019

exekias Nov 25, 2019

Choose a reason for hiding this comment

jsoriano Nov 25, 2019 • edited Loading

Choose a reason for hiding this comment

jsoriano Nov 25, 2019

Choose a reason for hiding this comment

kaiyan-sheng Nov 25, 2019

Choose a reason for hiding this comment

jsoriano Nov 25, 2019

Choose a reason for hiding this comment

mtojek commented Nov 20, 2019 •

edited

Loading

mtojek commented Nov 21, 2019 •

edited

Loading

mtojek commented Nov 22, 2019 •

edited

Loading

jsoriano Nov 25, 2019 •

edited

Loading