Skip to content

Latest commit

 

History

History
1548 lines (1548 loc) · 32.5 KB

awesome-hadoop.md

File metadata and controls

1548 lines (1548 loc) · 32.5 KB
Raw

Awesome Hadoop Awesome

A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources. Inspired by Awesome PHP , Awesome Python and Awesome Sysadmin

Hadoop

  • Apache Hadoop - Apache Hadoop
  • Apache Tez - A Framework for YARN-based, Data Processing Applications In Hadoop
  • SpatialHadoop - SpatialHadoop is a MapReduce extension to Apache Hadoop designed specially to work with spatial data.
  • GIS Tools for Hadoop - Big Data Spatial Analytics for the Hadoop Framework
  • Elasticsearch Hadoop - Elasticsearch real-time search and analytics natively integrated with Hadoop. Supports Map/Reduce, Cascading, Apache Hive and Apache Pig. ★ 701, pushed 128 days ago
  • dumbo - Python module that allows you to easily write and run Hadoop programs. ★ 966, pushed 138 days ago
  • hadoopy - Python MapReduce library written in Cython. ★ 224, pushed 240 days ago
  • mrjob - mrjob is a Python 2.5+ package that helps you write and run Hadoop Streaming jobs.
  • pydoop - Pydoop is a package that provides a Python API for Hadoop.
  • hdfs-du - HDFS-DU is an interactive visualization of the Hadoop distributed file system. ★ 139, pushed 1490 days ago
  • White Elephant - Hadoop log aggregator and dashboard ★ 171, pushed 1041 days ago
  • Kiji Project
  • Genie - Genie provides REST-ful APIs to run Hadoop, Hive and Pig jobs, and to manage multiple Hadoop resources and perform job submissions across them. ★ 462, pushed 130 days ago
  • Apache Kylin - Apache Kylin is an open source Distributed Analytics Engine from eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets
  • Crunch - Go-based toolkit for ETL and feature extraction on Hadoop ★ 130, pushed 655 days ago
  • Apache Ignite - Distributed in-memory platform

YARN

  • Apache Slider - Apache Slider is a project in incubation at the Apache Software Foundation with the goal of making it possible and easy to deploy existing applications onto a YARN cluster.
  • Apache Twill - Apache Twill is an abstraction over Apache Hadoop® YARN that reduces the complexity of developing distributed applications, allowing developers to focus more on their application logic.
  • mpich2-yarn - Running MPICH2 on Yarn ★ 77, pushed 481 days ago

NoSQL

Next Generation Databases mostly addressing some of the points: being non-relational, distributed, open-source and horizontally scalable.

  • Apache HBase - Apache HBase
  • Apache Phoenix - A SQL skin over HBase supporting secondary indices
  • happybase - A developer-friendly Python library to interact with Apache HBase. ★ 242, pushed 155 days ago
  • Hannibal - Hannibal is tool to help monitor and maintain HBase-Clusters that are configured for manual splitting. ★ 133, pushed 406 days ago
  • Haeinsa - Haeinsa is linearly scalable multi-row, multi-table transaction library for HBase ★ 122, pushed 158 days ago
  • hindex - Secondary Index for HBase ★ 335, pushed 609 days ago
  • Apache Accumulo - The Apache Accumulo™ sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • OpenTSDB - The Scalable Time Series Database
  • Apache Cassandra

SQL on Hadoop

SQL on Hadoop

  • Apache Hive - The Apache Hive data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL
  • Apache Phoenix A SQL skin over HBase supporting secondary indices
  • Apache HAWQ (incubating) - Apache HAWQ is a Hadoop native SQL query engine that combines the key technological advantages of MPP database with the scalability and convenience of Hadoop
  • Lingual - SQL interface for Cascading (MR/Tez job generator)
  • Cloudera Impala
  • Presto - Distributed SQL Query Engine for Big Data. Open sourced by Facebook.
  • Apache Tajo - Data warehouse system for Apache Hadoop
  • Apache Drill - Schema-free SQL Query Engine
  • Apache Trafodion

Data Management

Workflow, Lifecycle and Governance

Data Ingestion and Integration

DSL

  • Apache Pig - Apache Pig
  • Apache DataFu - A collection of libraries for working with large-scale data in Hadoop
  • vahara - Machine learning and natural language processing with Apache Pig ★ 51, pushed 993 days ago
  • packetpig - Open Source Big Data Security Analytics ★ 238, pushed 180 days ago
  • akela - Mozilla's utility library for Hadoop, HBase, Pig, etc. ★ 72, pushed 889 days ago
  • seqpig - Simple and scalable scripting for large sequencing data set(ex: bioinfomation) in Hadoop
  • Lipstick - Pig workflow visualization tool. Introducing Lipstick on A(pache) Pig ★ 359, pushed 268 days ago
  • PigPen - PigPen is map-reduce for Clojure, or distributed Clojure. It compiles to Apache Pig, but you don't need to know much about Pig to use it. ★ 419, pushed 133 days ago

Libraries and Tools

Realtime Data Processing

Distributed Computing and Programming

  • Apache Spark
  • Apache Crunch
  • Cascading - Cascading is the proven application development platform for building data applications on Hadoop.
  • Apache Flink - Apache Flink is a platform for efficient, distributed, general-purpose data processing.

  • Apache Apex (incubating) - Enterprise-grade unified stream and batch processing engine.

    Packaging, Provisioning and Monitoring

  • Apache Bigtop - Apache Bigtop: Packaging and tests of the Apache Hadoop ecosystem

  • Apache Ambari - Apache Ambari
  • Ganglia Monitoring System
  • ankush - A big data cluster management tool that creates and manages clusters of different technologies. ★ 19, pushed 504 days ago
  • Apache Zookeeper - Apache Zookeeper
  • Apache Curator - ZooKeeper client wrapper and rich ZooKeeper framework
  • Buildoop - Hadoop Ecosystem Builder ★ 27, pushed 348 days ago
  • Deploop - The Hadoop Deploy System
  • Jumbune - An open source MapReduce profiling, MapReduce flow debugging, HDFS data quality validation and Hadoop cluster monitoring tool.
  • inviso - Inviso is a lightweight tool that provides the ability to search for Hadoop jobs, visualize the performance, and view cluster utilization. ★ 122, pushed 268 days ago

Search

Search Engine Framework

  • Apache Nutch - Apache Nutch is a highly extensible and scalable open source web crawler software project.

Security

  • Apache Ranger - Ranger is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform.
  • Apache Sentry - An authorization module for Hadoop
  • Apache Knox Gateway - A REST API Gateway for interacting with Hadoop clusters.

Benchmark

  • Big Data Benchmark
  • HiBench ★ 358, pushed 138 days ago
  • Big-Bench ★ 30, pushed 186 days ago
  • hive-benchmarks ★ 2, pushed 819 days ago
  • hive-testbench - Testbench for experimenting with Apache Hive at any data scale. ★ 32, pushed 159 days ago
  • YCSB - The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare relative performance of NoSQL database management systems. ★ 1179, pushed 127 days ago

Machine learning and Big Data analytics

  • Apache Mahout
  • Oryx 2 - Lambda architecture on Spark, Kafka for real-time large scale machine learning ★ 777, pushed 134 days ago
  • MLlib - MLlib is Apache Spark's scalable machine learning library.
  • R - R is a free software environment for statistical computing and graphics.
  • RHadoop including RHDFS, RHBase, RMR2, plyrmr
  • RHive RHive, for launching Hive queries from R ★ 108, pushed 399 days ago
  • Apache Lens
  • Apache SINGA (incubating) - SINGA is a general distributed deep learning platform for training big deep learning models over large datasets

Misc.

Resources

Various resources, such as books, websites and articles.

Websites

Useful websites and articles

Presentations

Books

Hadoop and Big Data Events

Other Awesome Lists

Other amazingly awesome lists can be found in the awesome-awesomeness and awesome list.