-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
163 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,163 @@ | ||
| # Using the Apache Druid GUI to setup OpenTelemetry ingestion | ||
|
|
||
| This is a short, visual guide on how to use the [Apache Druid] GUI console to setup OpenTelemetry ingestion jobs using the | ||
| [druid-otlp-format extension](./README.md), developed by [mishmash io](https://mishmash.io). | ||
|
|
||
| To find more about the open source software contained in this repository - [click here.](../) | ||
|
|
||
| # Intro | ||
|
|
||
| At the time of writing the Druid GUI console does not directly allow configuring OTLP ingestion jobs. However, with a little 'hack', | ||
| it is possible. | ||
|
|
||
| Follow the steps below. | ||
|
|
||
| > [!WARNING] | ||
| > The guide here assumes you will be ingesting OpenTelemetry data published to Apache Kafka by the [Kafka Exporter.](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md) | ||
| > | ||
| > In other setups the steps might be different. | ||
| > | ||
| # Walk-through | ||
|
|
||
| #### 1. Open the console | ||
|
|
||
| Open the Druid GUI, select `Load data` from the top menu and then click `Streaming` in the dropdown: | ||
|
|
||
|  | ||
|
|
||
| #### 2. Start a new streaming spec | ||
|
|
||
| Click on the `Start a new streaming spec`: | ||
|
|
||
|  | ||
|
|
||
| #### 3. Select the source | ||
|
|
||
| When you start a new streaming spec you're given a choice to select where the data should come from. Click `Apache Kafka`: | ||
|
|
||
|  | ||
|
|
||
| #### 4. Connect to Kafka | ||
|
|
||
| To connect Druid to Kafka - enter your Kafka brokers and topic name in the right panel: | ||
|
|
||
|  | ||
|
|
||
| Click `Apply` when done. This will trigger Druid to connect and in a little while (if the connection was successful), Druid will | ||
| show some examples of what it read from the configured Kafka topic: | ||
|
|
||
|  | ||
|
|
||
| At this step - the data will be messy - Druid doesn't yet know how to interpret it. It's okay, click the `Next: Parse data` | ||
| button in the right panel. | ||
|
|
||
| #### 5. Apply the OTLP format | ||
|
|
||
| Now, you're given a choice of what `Input format` to apply on the incoming data: | ||
|
|
||
|  | ||
|
|
||
| Here's the tricky part - the `otlp` format is not available inside the `Input format` dropdown. You'll have to edit a bit of JSON | ||
| in order to get past this step. | ||
|
|
||
| On the bar just under the top menu, click on the final step - `Edit spec`. A JSON representation of the ingestion spec will be | ||
| shown to you. Edit the `inputFormat` JSON object: | ||
|
|
||
|  | ||
|
|
||
| Use one of the following JSONs: | ||
| - when ingesting `logs`: | ||
| ```json | ||
| ... | ||
| "inputFormat": { | ||
| "type": "otlp", | ||
| "otlpInputSignal": "logsRaw" | ||
| } | ||
| ... | ||
| ``` | ||
| - when ingesting `metrics`: | ||
| ```json | ||
| ... | ||
| "inputFormat": { | ||
| "type": "otlp", | ||
| "otlpInputSignal": "metricsRaw" | ||
| } | ||
| ... | ||
| ``` | ||
| - when ingesting `traces`: | ||
| ```json | ||
| ... | ||
| "inputFormat": { | ||
| "type": "otlp", | ||
| "otlpInputSignal": "tracesRaw" | ||
| } | ||
| ... | ||
| ``` | ||
|
|
||
| When done editing, just get back to the `Parse data` step, you don't need to click anywhere to confirm the changes you made. | ||
|
|
||
| Now, Druid will re-read the sampled Kafka messages and will parse them: | ||
|
|
||
|  | ||
|
|
||
| At this point - click the `Next: Parse time` button in the right panel to continue. | ||
|
|
||
| #### 6. Configure the data schema | ||
|
|
||
| Now, Druid needs to know a few more things about how you'd like your data to be organized. | ||
|
|
||
| Select which column will be used for its **time-partitioning.** For example, click on the `time_unix_nano` column header, | ||
| make sure it shows in the right bar, and that `Format` says `nano`. Then click 'Apply`: | ||
|
|
||
|  | ||
|
|
||
| When done, click `Next: Transform`: | ||
|
|
||
|  | ||
|
|
||
| Here you can configure optional transformations of the data before saving it inside Druid. Continue to the next step - `Filter`. | ||
|
|
||
|  | ||
|
|
||
| Filtering is also an optional step. Continue to the `Configure schema` step. | ||
|
|
||
| Here you can edit the table schema. In the bar on the right - turn off the `Explicitly specify schema` radio button: | ||
|
|
||
|  | ||
|
|
||
| Doing this will trigger a dialog to pop up, just confirm by clicking `Yes - auto detect schema`: | ||
|
|
||
|  | ||
|
|
||
| Then proceed to the `Partition` step. Select a `Primary partitioning (by time)`. In this example, we're setting it to `hour` | ||
| for hourly partitions: | ||
|
|
||
|  | ||
|
|
||
| Continue to `Tune`: | ||
|
|
||
|  | ||
|
|
||
| Set the `Use earliest offset` to `True` and continue to `Publish`: | ||
|
|
||
|  | ||
|
|
||
| See if you would like to rename the `Datasource name` or any of the other options, but the defaults should be okay. Move to the | ||
| next step - `Edit spec`: | ||
|
|
||
|  | ||
|
|
||
| This is the same step we went to earlier in order to set the `InputFormat` to `otlp`. This time - there's nothing to do here, | ||
| so, just click `Submit` in the right panel and that's it! | ||
|
|
||
| At the end, you should get a screen similar to this: | ||
|
|
||
|  | ||
|
|
||
| ##### Congratulations! :) | ||
|
|
||
| You can now start querying OpenTelemetry data! :) | ||
|
|
||
|
|
||
|
|