bgpalerter -t verification?

[ghenry]

Hi all,

I'm running the test option for fake alerts to verify emailing, but I can't see any attempts in the logs. Should I?

I'm using sendgrid and I wanted to validate that side, but can't see anything to indicate the fake alerts stage, let alone reporting?

Even with the placeholder config, I expected to see some logging to show attempts?

Lastly, fantastic documentation. I've had no issues installing, setting up cron for updates or systemd. Congratulations!

Thanks,
Gavin.

[massimocandela]

Hi @ghenry,

When you run the test, alerts should be sent to all the reports monitoring the hijack channel (see here).
So please, be sure you are monitoring somewhere the hijack channel.

By default, reportFile is enabled in config.yml.example, this means that with the default configuration, in logs/reports.log you will be able to see the reports. Additionally, you can enable reportPullAPI and hit http://localhost:8011 to see the alerts being collected. I just checked with both the source code and the binary version and it works with the default configuration.

Thanks for the nice words about the documentation!

[ghenry]

Hi @massimocandela

Here's my config. All I've uncommented is the reportEmail part of newPrefixes and then ran with -t and xxx are email address or a password/domain:

connectors:
  - file: connectorRIS
    name: ris
    params:
      carefulSubscription: true
      url: ws://ris-live.ripe.net/v1/ws/
      perMessageDeflate: true
      subscription:
        moreSpecific: true
        type: UPDATE
        host:
        socketOptions:
          includeRaw: false

#  - file: connectorRISDump
#    name: dmp

monitors:
  - file: monitorHijack
    channel: hijack
    name: basic-hijack-detection
    params:
      thresholdMinPeers: 3

  - file: monitorNewPrefix
    channel: newprefix
    name: prefix-detection
    params:
      thresholdMinPeers: 3

  - file: monitorPath
    channel: path
    name: path-matching
    params:
      thresholdMinPeers: 1

  - file: monitorVisibility
    channel: visibility
    name: withdrawal-detection
    params:
      thresholdMinPeers: 40

  - file: monitorAS
    channel: misconfiguration
    name: asn-monitor
    params:
      thresholdMinPeers: 3

  - file: monitorRPKI
    channel: rpki
    name: rpki-monitor
    params:
      thresholdMinPeers: 3
      checkUncovered: false
      checkDisappearing: false

  - file: monitorROAS
    channel: rpki
    name: rpki-diff
    params:
      enableDiffAlerts: true
      enableExpirationAlerts: true
      enableExpirationCheckTA: true
      enableDeletedCheckTA: true
      roaExpirationAlertHours: 2
      checkOnlyASns: true
      toleranceDeletedRoasTA: 20
      toleranceExpiredRoasTA: 20

  - file: monitorPathNeighbors
    channel: path
    name: path-neighbors
    params:
      thresholdMinPeers: 3

reports:
  - file: reportFile
    channels:
      - hijack
      - newprefix
      - visibility
      - path
      - misconfiguration
      - rpki
    params:
      persistAlertData: false
      alertDataDirectory: alertdata/

  - file: reportEmail
    channels:
      - hijack
      - newprefix
      - visibility
      - path
      - misconfiguration
      - rpki
    params:
      showPaths: 5 # Amount of AS_PATHs to report in the alert
      senderEmail: bgpalerter@xxx
      # BGPalerter uses nodemailer.
      # The smtp section can be configured with all the parameters available at https://nodemailer.com/smtp/
      # the following are just the most useful one
      smtp:
        host: smtp.sendgrid.net
        port: 465
        secure: true # If true the connection will use TLS when connecting to server. If false it will be still possible doing connection upgrade via STARTTLS
        ignoreTLS: false # If true TLS will be completely disabled, including STARTTLS. Set this to true if you see certificate errors in the logs.
        auth:
          user: apikey
          pass: xxx
      notifiedEmails:
        default:
          - xxx
        noc:
          - xxx
          - xxx

#  - file: reportSlack
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      showPaths: 0 # Amount of AS_PATHs to report in the alert
#      colors:
#        hijack: '#d60b1c'
#        newprefix: '#fa9548'
#        visibility: '#fad648'
#        path: '#42cbf5'
#        rpki: '#d892f0'
#      hooks:
#        default: _YOUR_SLACK_WEBHOOK_URL_
#        noc: _YOUR_SLACK_WEBHOOK_URL_

#  - file: reportKafka
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      host: localhost
#      port: 9092
#      topics:
#        default: bgpalerter

#  - file: reportSyslog
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - asn-monitor
#      - misconfiguration
#      - rpki
#    params:
#      host: 127.0.0.1
#      port: 514
#      transport: udp
#      templates: # See here how to write a template https://github.com/nttgin/BGPalerter/blob/main/docs/context.md
#        default: "++BGPalerter-3-${type}: ${summary}|${earliest}|${latest}"
#        hijack: "++BGPalerter-5-${type}: ${summary}|${prefix}|${description}|${asn}|${newprefix}|${neworigin}|${earliest}|${latest}|${peers}"
#        newprefix: "++BGPalerter-4-${type}: ${summary}|${prefix}|${description}|${asn}|${newprefix}|${neworigin}|${earliest}|${latest}|${peers}"
#        visibility: "++BGPalerter-5-${type}: ${summary}|${prefix}|${description}|${asn}|${earliest}|${latest}|${peers}"
#        misconfiguration: "++BGPalerter-3-${type}: ${summary}|${asn}|${prefix}|${earliest}|${latest}"

#  - file: reportAlerta
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      severity:
#        hijack: critical
#        newprefix: informational
#        visibility: debug
#        path: trace
#      resourceTemplates: # See here how to write a template https://github.com/nttgin/BGPalerter/blob/main/docs/context.md
#        default: "${type}"
#        hijack: "hijack::${prefix}@@${asn}"
#        newprefix: "newprefix::${prefix}@@${asn}"
#        visibility: "visibility::${prefix}@@${asn}"
#      urls:
#        default: _YOUR_ALERTA_API_URL_
#        noc: _YOUR_ALERTA_API_URL_

#  - file: reportWebex
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      hooks:
#        default: _YOUR_WEBEX_WEBHOOK_URL_
#        noc: _YOUR_WEBEX_WEBHOOK_URL_

#  - file: reportHTTP
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      templates: # See here how to write a template https://github.com/nttgin/BGPalerter/blob/main/docs/context.md
#        default: '{"text": "${summary}"}'
#      headers:
#      isTemplateJSON: true
#      showPaths: 0 # Amount of AS_PATHs to report in the alert
#      hooks:
#        default: _YOUR_WEBHOOK_URL_
#        noc: _YOUR_WEBHOOK_URL_

#  - file: reportTelegram
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      showPaths: 0 # Amount of AS_PATHs to report in the alert
#      botUrl: https://api.telegram.org/bot<_BOT_ID_>/sendMessage
#      chatIds:
#        default: _CHAT_ID_
#        noc: _CHAT_ID_

#  - file: reportPullAPI
#    channels:
#      - hijack
#      - newprefix
#      - visibility
#      - path
#      - misconfiguration
#      - rpki
#    params:
#      maxAlertsAmount: 25

############################
# Notification settings:
# - notificationIntervalSeconds
#       Defines the amount of seconds after which an alert can be repeated. An alert is repeated only if the event that
#       triggered it is not yet solved.
# - persistStatus
#       Persist the status of BGPalerter. If the process is restarted, the list of alerts already sent is recovered
#       and they are not repeated. The process must be able to write on disc, this option will create a file inside .cache/

notificationIntervalSeconds: 86400
persistStatus: true


############################
# REST API settings:
# - rest.host
#       The IP address the server will listen. The default value is localhost, this means the API will not be reachable
#       from another host. To make it public use null or 0.0.0.0.
# - rest.port
#       The port the server will listen

rest:
  host: localhost
  port: 8011

logging:
  directory: logs
  logRotatePattern: YYYY-MM-DD
  maxRetainedFiles: 10
  maxFileSizeMB: 15
  compressOnRotation: false
  useUTC: true
  
checkForUpdatesAtBoot: true
generatePrefixListEveryDays: 0

############################
# Process monitoring settings:
# Uncomment or add classes under processMonitors if you want to monitor or send logs about the status of the BGPalerter process

#processMonitors:
#  - file: uptimeApi
#    params:
#      useStatusCodes: true
#
#  - file: uptimeHealthcheck
#    params:
#      url: url_to_poll
#      intervalSeconds: 300
#      method: get
#
#  - file: sentryModule
#    params:
#      dsn: https://<key>@sentry.io/<project>


############################
# The files containing the monitored prefixes. Please see prefixes.yml for an example.
# This is an array (use new lines and dashes!)

monitoredPrefixesFiles:
  - prefixes.yml

############################
# The file containing the user groups.
# User groups can be specified
# 1) directly above, in each report module; or
# 2) inside an external file, as specified below (disabled by default).
# Using an external file allows BGPalerter to auto-reload the user group definitions
# when the external file is changed.

# groupsFile: groups.yml.example

############################
# HTTP proxy setting:
# Allow to run BGPalerter behind an HTTP/HTTPS proxy.
# You can also specify which module can bypass the proxy.
# More information here: https://github.com/nttgin/BGPalerter/blob/main/docs/http-proxy.md

# httpProxy: http://username:[email protected]:9000


############################
# RPKI settings:
# Global RPKI settings shared across all monitors requiring RPKI data
# More information here: https://github.com/nttgin/BGPalerter/blob/main/docs/rpki.md
#
# 29/06/2021: rpkiclient has been set as default vrpProvider in this example since is the only one supporting
# ROAs expiration data. See what expiration data enables you to do: https://github.com/nttgin/BGPalerter/releases/tag/v1.28.1

rpki:
  vrpProvider: rpkiclient
  preCacheROAs: true
  refreshVrpListMinutes: 15
  markDataAsStaleAfterMinutes: 120


############################
# Advanced settings (Don't touch here!)
# Please, refer to the documentation to know the meaning of the following parameters.

alertOnlyOnce: false
fadeOffSeconds: 360
checkFadeOffGroupsSeconds: 30
pidFile: bgpalerter.pid
maxMessagesPerSecond: 6000
multiProcess: false
environment: production
configVersion: 2

[ghenry]

Just to note, this was auto-generated config, I just uncommented the bits I've mentioned.

[ghenry]

Oh, and built using build.sh

[massimocandela]

Hi @ghenry,

I downloaded your config file -as is- including the redacted parameters, and it works. I can see many logs being generated inside logs/reports.log by reportFile, when I run the npm run serve -- --t or the binary. Ofc, the reportEmail part is not working due to the missing parameters, but that's not what we are checking now.

• Are you looking inside the correct log file?
• Is it possible that logs/error.log is reporting some other error (e.g., something specific for your machine) and it doesn't reach the test phase?
• Does your process have the rights to write the log files? Do you see any log file at all?
• Are you letting BGPalerter run long enough with the option -t? (e.g., 1 minute)

[ghenry]

Hi @massimocandela

Thanks. I've shut down the process and am running with the test option now. Logs are:

bgpalerter@daenerys:~$ tail -f logs/error.log 
2021-08-31T06:25:48+00:00 info: Test connector connected

I'll confirm back other things later, but that looks OK?

Gavin.

[ghenry]

So nothing new yet, still sitting at:

tail -f logs/error.log 
2021-08-31T06:25:48+00:00 info: Test connector connected

[massimocandela]

Something we can try:

Could you show the content of reports.log?

Could you be sure you did git pull in main and you have the last version? Also, do npm ci before to run it.

After, can you try to apply the config above, exactly as you posted it, including redacted parts. Just to check if the error is somewhere else in the config

[ghenry]

reports.log:

bgpalerter@daenerys:~$ ll logs/reports.log 
lrwxrwxrwx 1 bgpalerter bgpalerter 22 Aug 31 09:47 logs/reports.log -> reports-2021-08-31.log
bgpalerter@daenerys:~$ ll logs/reports-2021-08-31.log 
-rw-rw-r-- 1 bgpalerter bgpalerter 14512500 Aug 31 08:56 logs/reports-2021-08-31.log

tail logs/reports.log 
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,15563] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,204092] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:193.0.0.0/21 peer:1.2.3.5 path:[1,2,3,3333] nextHop:1.2.3.4 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:175.254.205.0/25 peer:124.0.0.3 path:[1,2,3,4321] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:170.254.205.0/25 peer:124.0.0.3 path:[1,2,3,4321] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:165.254.255.0/25 peer:124.0.0.2 path:[1,2,3,[4,15562]] nextHop:124.0.0.2 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884:ffff::/48 peer:124.0.0.3 path:[1,2,3,208585] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,[204092,45]] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,15563] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,204092] nextHop:124.0.0.3 aggregator:nul

I'm running upgrade.sh in cron, so should be latest, not the one I've built.

Will do.

[ghenry]

tail -f logs/error.log 
2021-08-31T06:25:48+00:00 info: Test connector connected
2021-08-31T08:35:26+00:00 info: Test connector connected
2021-08-31T08:35:52+00:00 info: Test connector connected
2021-08-31T08:36:34+00:00 info: ris connector connected
2021-08-31T08:40:00+00:00 info: Test connector connected
2021-08-31T08:47:36+00:00 info: Test connector connected
2021-08-31T08:50:14+00:00 info: Test connector connected

and

./bgpalerter-linux-x64 -t
Testing BGPalerter configuration. WARNING: remove -t option for production monitoring.
Loaded config: /home/bgpalerter/config.yml
BGPalerter, version: 1.28.2 environment: production

[ghenry]

ll logs/reports.log 
lrwxrwxrwx 1 bgpalerter bgpalerter 22 Aug 31 09:50 logs/reports.log -> reports-2021-08-31.log



tail logs/reports.log 
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,15563] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,204092] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:193.0.0.0/21 peer:1.2.3.5 path:[1,2,3,3333] nextHop:1.2.3.4 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:175.254.205.0/25 peer:124.0.0.3 path:[1,2,3,4321] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561907 prefix:170.254.205.0/25 peer:124.0.0.3 path:[1,2,3,4321] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:165.254.255.0/25 peer:124.0.0.2 path:[1,2,3,[4,15562]] nextHop:124.0.0.2 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884:ffff::/48 peer:124.0.0.3 path:[1,2,3,208585] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,[204092,45]] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,15563] nextHop:124.0.0.3 aggregator:null
2021-08-31T07:56:01+00:00 verbose: type:announcement timestamp:1630396561908 prefix:2a00:5884::/32 peer:124.0.0.3 path:[1,2,3,204092] nextHop:124.0.0.3 aggregator:null

[massimocandela]

Hi @ghenry,

I'm back on this. What you see in logs/reports.log are the alerts generated; hence, the process is trying to send alerts.
The issue is that no error logs are reported about the failed email deliveries. I investigated a bit on this and I found the reason for it. Essentially, reportEmail correctly tried to deliver emails to the default address when an email address for a user group was not declared. However, when the user group was not declared at all (which usually doesn't happen because BGPalerter generates the user groups by default), the delivery was not even attempted (and so no logs). This indeed made -t less useful for testing reportEmail configurations.

I tried now (version in the dev branch) with your configuration and in logs/error.log you will find some errors from sendgrid.net.

Thanks for reporting this. This will go out with the next release.

[ghenry]

Thanks Massimo. So all is working as expected and if not, at the moment I won't see an email send failure in the logs?

[massimocandela]

If you are using the source code in the main branch, you will not see email send errors while using the option -t. If you switch to the source code in the dev branch, I fixed it and you will be able to see all the email send errors, including authentication errors (e.g., to sendgrid.net). You can also download the binary of the new pre-release v1.28.3 which already includes this fix.

[massimocandela]

btw, all of this if you are interested in running the option -t to test the configuration. If you run it normally (without -t), with a configuration generated by BGPalerter, you will see errors in case the email cannot be sent independently from my fix above (but you have to wait for a real alert to happen).

[ghenry]

Ah, OK. So it's just the test flag that had the issue. Understood.

[massimocandela]

released in v1.28.3

[ghenry]

Thanks. We know it's working in production. Just got an alert about one of the monitored ASNs. Thanks for this great project!