Bugzilla ETL

Notice: This ETL is no longer used - active development has moved to https://github.com/klahnakoski/Bugzilla-ETL.

A set of Pentaho DI jobs to extract bug versions from a bugzilla database and store them in an elasticsearch index. This ETL drives dashboards for BMO, for various teams at Mozilla Corporation.

Requirements

• an elasticsearch cluster where you can CRUD the index bugs
• a working PDI (a.k.a kettle) installation (free community edition should work fine). Tested with PDI CE 4.3

Minimal instructions

• Clone this project into a local directory

• Configure the elasticsearch indexes (put a cluster node in place of localhost):

  • Optionally: clean out previous indexes:

    curl -XDELETE 'http://localhost:9200/bugs'

  • Initialize the elasticsearch mappings:

    curl -XPOST 'http://localhost:9200/bugs' --data @configuration/es/bug_version.json

• Configure Pentaho DI:

  • add a directory .kettle in your $KETTLE_HOME
  • there, create a file kettle.properties
  • in that file, add settings for bugs_db_host, bugs_db_port, bugs_db_user, bugs_db_pass and bugs_db_name for your bugzilla-database connection.
  • add settings for ES_NODES, ES_CLUSTER, ES_INDEX

• If necessary, modify bin/import_bugs.sh, then run it to import the full data set.

• Later on, use bin/update_bugs_incr.sh to read incremental modifications from the MySQL database

Known issues

• Some cases where a user's bugzilla ID changes mid-history for a bug can't be handled automatically, and should be added to configuration/kettle/bugzilla_aliases.txt. There are several alias-related scripts and transformations that help to detect these types of changes. See bin/find_aliases.sh, bin/find_all_aliases.sh, transformations/find_aliases.ktr, and transformations/detect_new_aliases.ktr.
• Mozilla Bug 804946 causes some trouble with the ETL. See Bug 804961 for details.