Integrate with taskcluster

[mostlygeek]

Current buildhub has to scrape (download archives, decompress, extract metadata) to build its database. Scraping sucks. Better would be to download a JSON file full of the metadata we need.

Since the migration from buildbot/older systems to Taskcluster is nearly complete (90% by end of Feb), we should explore integrating with TC more closely.

I propose that we work with @kmoir's Build and VCS team and add two new tasks to the Firefox Task graph:

1. A task that creates a JSON document with the necessary metadata
2. A task that notifies Buildhub an upload is available (optional)

The first task, a JSON document will live in archive.mozilla.org with the builds. This way archive is continues to be the canonical source of metadata. It's just in an easier to extract and consume.

The second task is optional. It is an explicit TC task to publish the metadata into Buildhub. I think it should be optional because failure to contact Buildhub should not fail the build. We can also accomplish this today with S3 events, SQS and lambda. However, catlee made a good point that a TC task would be more visible/easier to debug. More discussion required here.

Other things that require some design:

• JSON document $schema and process for change control
• Latency of indexing, we've seen it < 5seconds with s3+lambda events
• What to build

[mostlygeek]

@leplatrem can you link / provide the current JSON schema for data in Buildhub?

[Natim]

@mostlygeek Here you go: https://github.com/mozilla-services/buildhub/blob/master/jobs/buildhub/initialization.yml#L71-L198

[mostlygeek]

This is being worked on in #437 and #465 closing this in favor of those issues.