Maintaining updates between servers

[zml2008]

Many networks run multiple servers sharing permissions data. This works fine until a change needs to be made and all the servers have to be reloaded.

Situations here:

• multiple servers, one proxy
  • proxy as sync server/message broker
  • proxy can act as host for webui as well?
  • can we make configuration automatic?
• multiple servers, multiple proxies
  • separate server, or other message queueing software? (redis, zeromq, etc)
  • manual configuration required

Potential links:

• Apache QPID

[egg82]

networking is hard. Servers go down, IPs change, VLANs vanish..
One of the better solutions I've seen is redundancy. The more, the better.

An example is using RabbitMQ as a pub/sub, and Redis as a pub/sub backup for Rabbit. Hell, you can even throw an SQL table in there if you feel like a little masochism that day. Push messages through both. Attach an ID to a message and push it through both systems, then just check the message ID on the other side against a "recently-received" to ensure you're not accidentally duplicating messages.

Additionally, Redis can act as a datastore similar to SQL. Cache upwards; fetch a value from SQL, then put it in Redis, then to an in-memory cache on the server. Record bad/missing data so you don't try to fetch over and over again, and solved. Mostly.

Implementation detail is the biggest bitch. Create a List<MessageSystem> and a List<StorageSystem> (or similar) and have your storage/messaging implement a common interface to make things easier. Run through the list when you want to fetch/store data. Learned that one the hard way, and it's still costing me a bunch of time. Coincidentally, this will make it easier to implement new storage/messaging systems later on when you discover more that your users will inevitably want because of fucking course they do.