Home

Gobblin is an open source unified data ingestion framework to bring significant amount of data from internal data sources (data generated on premise) and external data sources (data sourced from external web sites) into one central repository (HDFS) for analysis.

As companies are moving more and more towards a data-driven decision making business model, increasing number of business products are driven by business insights from data generated both internally on premise or sourced externally from public web sites or web services. Gobblin is developed to address ingesting those big data with ease:

• Centralized data lake: standardized data formats, directory layouts;
• Standardized catalog of lightweight transformations: security filters, schema evolution, type conversion, etc;
• Data quality measurements and enforcement: schema validation, data audits, etc;
• Scalable ingest: auto-scaling, fault-tolerance, etc;
• Ease of operations: centralized monitoring, enforcement of SLA-s etc;
• Ease of use: self-serve on-boarding of new datasets to minimize time and involvement of engineers.