-
Notifications
You must be signed in to change notification settings - Fork 265
Home
Summingbird is a library that lets you write MapReduce programs that look like native Scala or Java collection transformations. So, while a word-counting aggregation in pure Scala might look like this:
def wordCount(source: Iterable[String], store: MutableMap[String, Long]) =
source.flatMap { sentence =>
toWords(sentence).map(_ -> 1L)
}.foreach { case (k, v) => store.update(k, store.get(k) + v) }Counting words in Summingbird looks like this:
def wordCount[P <: Platform[P]]
(source: Producer[P, String], store: P#Store[String, Long]) =
source.flatMap { sentence =>
toWords(sentence).map(_ -> 1L)
}.sumByKey(store)The logic is exactly the same, and the code is almost the same. The main difference is that you can execute the Summingbird program in “batch mode” (using Scalding), in “realtime mode” (using Storm), or on both Scalding and Storm in a hybrid batch/realtime mode that offers your application very attractive fault-tolerance properties.
Summingbird provides you with the primitives you need to build rock solid production systems.
- Example hybrid summingbird github project.
- Open Sourcing Announcement, Twitter Eng Blog
- Introduction to Summingbird (by @posco)
- LambdaJam 2013 Summingbird Workshop (by @sritchie)
- Summingbird: StreamingMapReduce at Twitter (Sam’s talk from the AK Data Science Summit)
- CUFP 2013: Realtime MapReduce at Twitter (by @sritchie)
- Boston Storm Users: Summingbird, Scala & Storm (slides)
- PNW Scala 2013: Taking Hadoop Realtime with Summingbird
- The Road to Summingbird: Stream Processing at (Every) Scale
We’re very excited about Summingbird’s development as we move beyond our initial release. Some of our future plans include:
- Support for more execution platforms (Spark and Akka seem like clear candidates)
- Pluggable optimizations for the
Producergraph layer- Projection and filter pushdown
- Support for filter-aware data sources, like Parquet
- Libraries of higher-level mathematics and machine learning code on top of Summingbird’s
Producerprimitives - More extensions to Summingbird’s related projects (listed below)
- More tutorials and examples with public data sources
More information on these issues and many more can be found on the Summingbird issue tracker.
Discussion about development and new features occurs primarily on the Summingbird mailing list. To join the mailing list, email mailto:[email protected]. The same address, [email protected], is used for posting once you’ve joined.
Issues should be reported on the GitHub issue tracker. Simpler issues appropriate for first-time contributors looking to help out are tagged “newbie”. To see which issues are slated for a planned release, check out the milestones page.
Follow @summingbird on Twitter for updates.
Do you use Summingbird? Please contact us so we can add you to the Powered By wiki page.
The Summingbird projects spawned a number of related subprojects, notably:
Algebird is an abstract algebra library for Scala. Many of the data structures included in Algebird have Monoid implementations, making them ideal to use as values in Summingbird aggregations.
Summingbird uses the Bijection project’s Injection typeclass to share serialization between different execution platforms and clients.
Summingbird’s Storm and Scalding platforms both use the Kryo library for serialization. Chill augments Kryo with a number of helpful configuration options, and provides modules for use with Storm, Scala, Hadoop. Chill is also used by the Berkeley Amp Lab’s Spark project.
Tormenta provides a type-safe layer over Storm’s Scheme and Spout interfaces.
Summingbird’s client is implemented using Storehaus’s async key-value store traits. The Storm platform makes use of Storehaus’s MergeableStore trait to perform real-time aggregations into a number of commonly used backing stores, including Memcached and Redis.