Remove Zookeeper dependency

[jdegoes]

Zookeeper is a large, complex piece of software which requires installation and maintenance. There are many ways for Zookeeper to fail, and Zookeeper requires ongoing management in order to ensure continuous uptime.

While Zookeeper is very appropriate for a large scale distributed storage and analysis system, it is less appropriate for a single node analytics engine like Precog. The dependency just needlessly complicates the architecture, adds more points of failure, results in more lines of code that have to be maintained, and makes Precog more difficult for non-IT wizards to deploy.

In order to simplify the number of moving pieces in Precog, Zookeeper needs to be eliminated as a dependency. Once Kafka is removed (see #524), extraction of Zookeeper will be trivial as Zookeeper is used exclusively for tracking offsets.

[nuttycom]

See description of how to produce an environment that is not dependent upon
Kafka - Zookeeper is only used for tracking offsets in Kafka for recovery
purposes, so removing Kafka obviates the need for Zookeeper entirely.

On Sat, Oct 5, 2013 at 4:13 PM, John A. De Goes [email protected]:

Zookeeper is a large, complex piece of software which requires
installation and maintenance. There are many ways for Zookeeper to fail,
and Zookeeper requires ongoing management in order to ensure continuous
uptime.

While Zookeeper is very appropriate for a large scale distributed storage
and analysis system, it is less appropriate for a single node analytics
engine like Precog. The dependency just needlessly complicates the
architecture, adds more points of failure, results in more lines of code
that have to be maintained, and makes Precog more difficult for non-IT
wizards to deploy.

In order to simplify the number of moving pieces in Precog, Zookeeper
needs to be eliminated as a dependency. Once Kafka is removed (see #524#524),
extraction of Zookeeper should be a fairly straightforward exercise.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/525
.