Hot reloading of configuration

[DamianZaremba]

Describe what you would like to achieve
Currently the configuration is only read on startup, changing the monitored prefixes requires restarting the instance.

In an environment which has many prefix monitors, the configuration of which change in a relatively dynamic manner this requires restarting the process very regularly (hourly/daily) interval.

Describe why the current solution (if any) is not satisfactory

1. During the process restart any RIS messages are lost, potentially causing missed messages
2. alarmOnlyOnce only applies during the single run, causing messages to re-trigger at the restart interval
3. notificationIntervalSeconds only applies during the single run, causing messages to re-trigger at the smaller of notificationIntervalSeconds and the restart interval

Provide an example
Start the instance with a prefixes.yml;

23.235.32.0/24:
    asn: [54113]
    description:  example prefix 1
    ignore: false
    ignoreMorespecifics: false

Once running update prefixes.yml;

23.235.32.0/24:
    asn: [54113]
    description:  example prefix 2
    ignore: false
    ignoreMorespecifics: false

23.235.36.0/24:
    asn: [54113]
    description:  example more specific prefix
    ignore: false
    ignoreMorespecifics: false

No new prefixes will be subscribed to, any withdrawals, announcements etc will not be notified.

Your information
@DamianZaremba, Fastly / Infra Bits

[DamianZaremba]

I think there are generally 2 approaches to this problem;

1. Implement hot re-configuration and support the relevant subscribing/unsubscribing in each connector & monitor filters.

2. Implement persisting alert state externally (either at runtime, or during a clean shutdown) and loading alert state on startup, reducing the impact of restarting on alerting. This could also be useful for things such as log/notify also on restore #60.

It doesn't touch the issue of potentially dropping messages, however there are numerous other cases (reconnects) where that could happen, so I think that could be treated separately.

Feedback on the idea and proposed options would be useful!

[stale]

This issue has been automatically closed as stale. This mechanism helps to prioritize feature requests which received more support from the community. If you want to open again this issue you have to provide a Pull Request.

[massimocandela]

Hi @DamianZaremba,

I implemented your proposal n.2 here: #273. It will probably go out with the next release. The status of the application is persisted now.

Thanks for your feedback.

[DamianZaremba]

That looks great, thanks for taking care of this @massimocandela!

[massimocandela]

Hi Damian, in the end I implemented also a hot reloading in case of changes in prefixes.yml
This issue has been opened, closed, re-opened, re-closed... 😛

[DamianZaremba]

That's great.

I did notice a side effect of 'restart while persisting alerts' being that the new release notifications are either super spammy (notify on startup) or never triggered (if restarting faster than 5 days)... so this is hopefully a better solution for my setup :)

[massimocandela]

Hey Damian, could you elaborate?
In particular I didn't get the part "the new release notifications are either super spammy".
I'm interested in understanding if there is something I can improve.

Thanks

Ciao,
Massimo

[DamianZaremba]

Hey, sure;

When you have a configuration with checkForUpdatesAtBoot: true and a newer release is available, the pending update is notified at every startup; in the case that the service is restarted every hour for configuration updates then you get a pending update notification every hour (which can be more regular than prefix notifications, causing a lot of noise).

When checkForUpdatesAtBoot: false is configured the hard coded time of 5 days is used (https://github.com/nttgin/BGPalerter/blob/master/src/connectors/connectorSwUpdates.js#L75) which will never be hit if the daemon is restarted at an interval of less than 5 days (or at least this doesn't appear to get persisted with the persistAlertData option).

For my specific use case disabling the checkForUpdatesAtBoot and accepting the connectorSwUpdates connector was then essentially redundant was acceptable given the desire for up to date prefixes.

Now there is the option of hot reloading I'm trying the current dev branch with checkForUpdatesAtBoot: true re-enabled which I expect will give me both software updates after 5 days & updated prefixes.

Hopefully that makes sense.