Add Google Big Query Exporter to Telemetry

[theluckiestsoul]

Component(s)

No response

Is your feature request related to a problem? Please describe.

I'm always frustrated when I have to manually export and transfer telemetry data to Google Big Query for analysis, which leads to delays and inefficiencies in our data analysis pipeline. Currently, our telemetry system lacks a direct exporter to Google Big Query, forcing us to rely on cumbersome workarounds, such as exporting data to CSV files and then uploading them to Big Query. This manual process is time-consuming, error-prone, and hinders our ability to quickly gain insights from our telemetry data.
By adding a Google Big Query exporter to the telemetry system, we can streamline our data analysis workflow, reduce manual labor, and unlock more timely and accurate insights.

Describe the solution you'd like

I would like to see a Google Big Query exporter integrated into our telemetry system, allowing us to:

1. Automatically export telemetry data to Google Big Query in real-time or at scheduled intervals.
2. Configure the exporter to select specific telemetry data streams, filter data, and define data formats (e.g., JSON or Avro).
3. Authenticate using a service account or API key to secure data transfer.
4. Monitor export status, errors, and data transfer metrics through the telemetry system's UI or API.
5. Support for partitioning data in Big Query based on date, device ID, or other relevant fields.

This solution would enable us to:

• Streamline our data analysis pipeline
• Reduce manual labor and errors
• Increase data freshness and accuracy
• Unlock more timely and insightful analytics
• Leverage Google Big Query's scalability and analytics capabilities

By implementing this solution, we can enhance our telemetry system's capabilities and improve our overall data analysis workflow.

Describe alternatives you've considered

Here are some alternatives I've considered:

1. Custom Scripting
2. Cloud Function
3. Manual Export and Upload

Additional context

No response

[atoulme]

@theluckiestsoul would you implement the exporter?

[theluckiestsoul]

@atoulme Thanks for considering my issue! I'd be glad to take on the implementation. I'll get started and make sure to follow any guidelines. Let me know if there’s anything specific you’d like included.