-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Metricbeat] add a network_summary metricset #15196
Merged
fearful-symmetry
merged 24 commits into
elastic:master
from
fearful-symmetry:add-network-summary
Jan 13, 2020
Merged
Changes from 20 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
d00d6db
initial commit of system/network_summary
fearful-symmetry 2c37aa4
add tests, clean up
fearful-symmetry 27d36de
update notice
fearful-symmetry 61e394d
update golang.org/x/sys
fearful-symmetry f7ac244
update notice
fearful-symmetry 0568dd5
fix change in sid.String
fearful-symmetry 2f8cf0a
try and revert vendor to master
fearful-symmetry 8389e4a
try and revert vendor to upstream/master
fearful-symmetry d705aa5
reset vendor from master
fearful-symmetry cf725ea
re-update go-sysinfo
fearful-symmetry 5d7fc72
Merge remote-tracking branch 'upstream/master' into add-network-summary
fearful-symmetry d1df761
Merge remote-tracking branch 'upstream/master' into add-network-summary
fearful-symmetry ae561f5
update fields
fearful-symmetry 83f803a
cleanup, updates
fearful-symmetry b444cab
make update
fearful-symmetry f55fea2
change mapping
fearful-symmetry 0e4f7aa
update tests
fearful-symmetry 6e92d88
remove change to winlogbeat for x/sys compat issues
fearful-symmetry 0ab13db
remove extra spaces
fearful-symmetry 699bd31
update fields
fearful-symmetry 9d6d91d
update mapping to use uint64
fearful-symmetry 73cbc2e
Merge remote-tracking branch 'upstream/master' into add-network-summary
fearful-symmetry b970bf4
update changelog
fearful-symmetry 9c30581
update notice
fearful-symmetry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| //// | ||
| This file is generated! See scripts/mage/docs_collector.go | ||
| //// | ||
|
|
||
| [[metricbeat-metricset-system-network_summary]] | ||
| === System network_summary metricset | ||
|
|
||
| beta[] | ||
|
|
||
| include::../../../module/system/network_summary/_meta/docs.asciidoc[] | ||
|
|
||
|
|
||
| ==== Fields | ||
|
|
||
| For a description of each field in the metricset, see the | ||
| <<exported-fields-system,exported fields>> section. | ||
|
|
||
| Here is an example document generated by this metricset: | ||
|
|
||
| [source,json] | ||
| ---- | ||
| include::../../../module/system/network_summary/_meta/data.json[] | ||
| ---- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
245 changes: 245 additions & 0 deletions
245
metricbeat/module/system/network_summary/_meta/data.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,245 @@ | ||
| { | ||
| "@timestamp": "2017-10-12T08:05:34.853Z", | ||
| "event": { | ||
| "dataset": "system.network_summary", | ||
| "duration": 115000, | ||
| "module": "system" | ||
| }, | ||
| "metricset": { | ||
| "name": "network_summary", | ||
| "period": 10000 | ||
| }, | ||
| "service": { | ||
| "type": "system" | ||
| }, | ||
| "system": { | ||
| "network_summary": { | ||
| "icmp": { | ||
| "InAddrMaskReps": 0, | ||
| "InAddrMasks": 0, | ||
| "InCsumErrors": 0, | ||
| "InDestUnreachs": 835, | ||
| "InEchoReps": 0, | ||
| "InEchos": 1, | ||
| "InErrors": 20, | ||
| "InMsgs": 836, | ||
| "InParmProbs": 0, | ||
| "InRedirects": 0, | ||
| "InSrcQuenchs": 0, | ||
| "InTimeExcds": 0, | ||
| "InTimestampReps": 0, | ||
| "InTimestamps": 0, | ||
| "InType3": 835, | ||
| "InType8": 1, | ||
| "OutAddrMaskReps": 0, | ||
| "OutAddrMasks": 0, | ||
| "OutDestUnreachs": 940, | ||
| "OutEchoReps": 1, | ||
| "OutEchos": 0, | ||
| "OutErrors": 0, | ||
| "OutMsgs": 941, | ||
| "OutParmProbs": 0, | ||
| "OutRedirects": 0, | ||
| "OutSrcQuenchs": 0, | ||
| "OutTimeExcds": 0, | ||
| "OutTimestampReps": 0, | ||
| "OutTimestamps": 0, | ||
| "OutType0": 1, | ||
| "OutType3": 940 | ||
| }, | ||
| "ip": { | ||
| "DefaultTTL": 64, | ||
| "ForwDatagrams": 33389, | ||
| "Forwarding": 1, | ||
| "FragCreates": 0, | ||
| "FragFails": 0, | ||
| "FragOKs": 0, | ||
| "InAddrErrors": 19, | ||
| "InBcastOctets": 431773621, | ||
| "InBcastPkts": 1686995, | ||
| "InCEPkts": 0, | ||
| "InCsumErrors": 0, | ||
| "InDelivers": 38037698, | ||
| "InDiscards": 0, | ||
| "InECT0Pkts": 3160941, | ||
| "InECT1Pkts": 0, | ||
| "InHdrErrors": 0, | ||
| "InMcastOctets": 0, | ||
| "InMcastPkts": 0, | ||
| "InNoECTPkts": 129521210, | ||
| "InNoRoutes": 0, | ||
| "InOctets": 167733588581, | ||
| "InReceives": 41266648, | ||
| "InTruncatedPkts": 0, | ||
| "InUnknownProtos": 0, | ||
| "OutBcastOctets": 0, | ||
| "OutBcastPkts": 0, | ||
| "OutDiscards": 1, | ||
| "OutMcastOctets": 0, | ||
| "OutMcastPkts": 0, | ||
| "OutNoRoutes": 4, | ||
| "OutOctets": 75001958794, | ||
| "OutRequests": 29261852, | ||
| "ReasmFails": 0, | ||
| "ReasmOKs": 0, | ||
| "ReasmOverlaps": 0, | ||
| "ReasmReqds": 0, | ||
| "ReasmTimeout": 0 | ||
| }, | ||
| "tcp": { | ||
| "ActiveOpens": 6172, | ||
| "ArpFilter": 0, | ||
| "AttemptFails": 1488, | ||
| "BusyPollRxPackets": 0, | ||
| "CurrEstab": 9, | ||
| "DelayedACKLocked": 111, | ||
| "DelayedACKLost": 1587, | ||
| "DelayedACKs": 516004, | ||
| "EmbryonicRsts": 2, | ||
| "EstabResets": 404, | ||
| "IPReversePathFilter": 0, | ||
| "InCsumErrors": 0, | ||
| "InErrs": 4, | ||
| "InSegs": 38005498, | ||
| "ListenDrops": 0, | ||
| "ListenOverflows": 0, | ||
| "LockDroppedIcmps": 0, | ||
| "MaxConn": -1, | ||
| "OfoPruned": 0, | ||
| "OutOfWindowIcmps": 39, | ||
| "OutRsts": 10739, | ||
| "OutSegs": 66991971, | ||
| "PAWSActive": 0, | ||
| "PAWSEstab": 19, | ||
| "PFMemallocDrop": 0, | ||
| "PassiveOpens": 1411, | ||
| "PruneCalled": 0, | ||
| "RcvPruned": 0, | ||
| "RetransSegs": 15996, | ||
| "RtoAlgorithm": 1, | ||
| "RtoMax": 120000, | ||
| "RtoMin": 200, | ||
| "SyncookiesFailed": 2, | ||
| "SyncookiesRecv": 0, | ||
| "SyncookiesSent": 0, | ||
| "TCPACKSkippedChallenge": 2, | ||
| "TCPACKSkippedFinWait2": 0, | ||
| "TCPACKSkippedPAWS": 1, | ||
| "TCPACKSkippedSeq": 4, | ||
| "TCPACKSkippedSynRecv": 8, | ||
| "TCPACKSkippedTimeWait": 0, | ||
| "TCPAbortFailed": 0, | ||
| "TCPAbortOnClose": 321, | ||
| "TCPAbortOnData": 2613, | ||
| "TCPAbortOnLinger": 0, | ||
| "TCPAbortOnMemory": 0, | ||
| "TCPAbortOnTimeout": 84, | ||
| "TCPAckCompressed": 232116, | ||
| "TCPAutoCorking": 162, | ||
| "TCPBacklogCoalesce": 509441, | ||
| "TCPBacklogDrop": 0, | ||
| "TCPChallengeACK": 2, | ||
| "TCPDSACKIgnoredNoUndo": 4623, | ||
| "TCPDSACKIgnoredOld": 242, | ||
| "TCPDSACKOfoRecv": 2, | ||
| "TCPDSACKOfoSent": 15, | ||
| "TCPDSACKOldSent": 1668, | ||
| "TCPDSACKRecv": 8205, | ||
| "TCPDSACKUndo": 36, | ||
| "TCPDeferAcceptDrop": 0, | ||
| "TCPDelivered": 54733743, | ||
| "TCPDeliveredCE": 0, | ||
| "TCPFastOpenActive": 0, | ||
| "TCPFastOpenActiveFail": 0, | ||
| "TCPFastOpenBlackhole": 0, | ||
| "TCPFastOpenCookieReqd": 0, | ||
| "TCPFastOpenListenOverflow": 0, | ||
| "TCPFastOpenPassive": 0, | ||
| "TCPFastOpenPassiveFail": 0, | ||
| "TCPFastRetrans": 5324, | ||
| "TCPFromZeroWindowAdv": 977, | ||
| "TCPFullUndo": 3, | ||
| "TCPHPAcks": 4648338, | ||
| "TCPHPHits": 22005367, | ||
| "TCPHystartDelayCwnd": 3535, | ||
| "TCPHystartDelayDetect": 24, | ||
| "TCPHystartTrainCwnd": 4609, | ||
| "TCPHystartTrainDetect": 205, | ||
| "TCPKeepAlive": 312560, | ||
| "TCPLossFailures": 1, | ||
| "TCPLossProbeRecovery": 387, | ||
| "TCPLossProbes": 12748, | ||
| "TCPLossUndo": 437, | ||
| "TCPLostRetransmit": 1499, | ||
| "TCPMD5Failure": 0, | ||
| "TCPMD5NotFound": 0, | ||
| "TCPMD5Unexpected": 0, | ||
| "TCPMTUPFail": 0, | ||
| "TCPMTUPSuccess": 0, | ||
| "TCPMemoryPressures": 0, | ||
| "TCPMemoryPressuresChrono": 0, | ||
| "TCPMinTTLDrop": 0, | ||
| "TCPOFODrop": 0, | ||
| "TCPOFOMerge": 13, | ||
| "TCPOFOQueue": 266342, | ||
| "TCPOrigDataSent": 54724129, | ||
| "TCPPartialUndo": 145, | ||
| "TCPPureAcks": 6490261, | ||
| "TCPRcvCoalesce": 5762438, | ||
| "TCPRcvCollapsed": 0, | ||
| "TCPRcvQDrop": 0, | ||
| "TCPRenoFailures": 0, | ||
| "TCPRenoRecovery": 0, | ||
| "TCPRenoRecoveryFail": 0, | ||
| "TCPRenoReorder": 0, | ||
| "TCPReqQFullDoCookies": 0, | ||
| "TCPReqQFullDrop": 0, | ||
| "TCPRetransFail": 0, | ||
| "TCPSACKDiscard": 123, | ||
| "TCPSACKReneging": 0, | ||
| "TCPSACKReorder": 2707, | ||
| "TCPSYNChallenge": 4, | ||
| "TCPSackFailures": 5, | ||
| "TCPSackMerged": 13758, | ||
| "TCPSackRecovery": 241, | ||
| "TCPSackRecoveryFail": 3, | ||
| "TCPSackShiftFallback": 1455, | ||
| "TCPSackShifted": 37107, | ||
| "TCPSlowStartRetrans": 2, | ||
| "TCPSpuriousRTOs": 3, | ||
| "TCPSpuriousRtxHostQueues": 16, | ||
| "TCPSynRetrans": 346, | ||
| "TCPTSReorder": 732, | ||
| "TCPTimeWaitOverflow": 0, | ||
| "TCPTimeouts": 2178, | ||
| "TCPToZeroWindowAdv": 977, | ||
| "TCPWantZeroWindowAdv": 109048, | ||
| "TCPWinProbe": 0, | ||
| "TCPZeroWindowDrop": 0, | ||
| "TW": 1443, | ||
| "TWKilled": 0, | ||
| "TWRecycled": 0 | ||
| }, | ||
| "udp": { | ||
| "IgnoredMulti": 0, | ||
| "InCsumErrors": 0, | ||
| "InDatagrams": 31359, | ||
| "InErrors": 0, | ||
| "NoPorts": 61, | ||
| "OutDatagrams": 31756, | ||
| "RcvbufErrors": 0, | ||
| "SndbufErrors": 0 | ||
| }, | ||
| "udpLite": { | ||
| "IgnoredMulti": 0, | ||
| "InCsumErrors": 0, | ||
| "InDatagrams": 0, | ||
| "InErrors": 0, | ||
| "NoPorts": 0, | ||
| "OutDatagrams": 0, | ||
| "RcvbufErrors": 0, | ||
| "SndbufErrors": 0 | ||
| } | ||
| } | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| The System `network_summary` metricset provides network IO metrics collected from the | ||
| operating system. These events are global and sorted by protocol. | ||
|
|
||
| This metricset is available on: | ||
|
|
||
| - Linux |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it common practice in metricbeat to use camel case for a set of 'undocumented' metrics? Do we want to convert to snake case (e.g. github.com/stoewer/go-strcase).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, my thinking was that since we're inserting "blind" metrics with no real opinionated manipulating of the data, keeping the key values the same as the their OS values might be a good idea. Let me hunt around the system module to see if we do this in other places.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@exekias @jsoriano Do we have any precedence on using Pascal/CamelCase in events in metricbeat?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have done this when ingesting metrics in raw from a source, to avoid confusion. AWS cloudwatch is an example. That said, I don't have a strong opinion here