Skip to content
This repository has been archived by the owner on Dec 26, 2024. It is now read-only.

Commit

Permalink
#23: Added demo 3 to README
Browse files Browse the repository at this point in the history
  • Loading branch information
@bednar
bednar committed Jul 29, 2019
1 parent a071a11 commit 3ce71f7
Show file tree
Hide file tree
Showing 5 changed files with 107 additions and 66 deletions.
39 changes: 39 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -536,6 +536,45 @@ This example show how to store NiFi Records as a LineProtocol into multiple envi
<img src="assets/doc/demo3-setwriter.png" height="250px">
### Expose data from InfluxDB 2.0 on the particular HTTP endpoint.
This example show how to exposing InfluxDB 2.0 data by NiFi.
#### GetInfluxDatabase_2
The processor is configured to invoke static flux query:
```
from(bucket: "my-bucket")
|> range(start: 0)
|> filter(fn: (r) => r._measurement == "tweets")
|> drop(columns: ["keyword", "lang", "user_verified"])
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> limit(n:1)
|> keep(columns: ["tweet_id", "screen_name", "text"])
```
The result is mapped as CSV and returned as a response to incoming HTTP request.
<img src="assets/doc/demo4-static-query.png" height="250px">
#### GetInfluxDatabaseRecord_2
The processor invoke a flux query that is pass as a http query parameter:
```bash
curl -i -X GET -G http://localhost:8234 \
--data-urlencode 'accept=xml' \
--data-urlencode 'query=from(bucket: "my-bucket")
|> range(start: 0) |> filter(fn: (r) => r._measurement == "docker_container_status")
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> limit(n:10, offset: 0)'
```
The result is mapped to format that is specified in request `accept` parameter.
<img src="assets/doc/demo4-dynamic-query.png" height="250px">
## Contributing
If you would like to contribute code you can do through GitHub by forking the repository and sending a pull request into the `master` branch.
Expand Down
Binary file added assets/doc/demo4-dynamic-query.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/doc/demo4-static-query.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
130 changes: 65 additions & 65 deletions scripts/flow.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,40 +27,38 @@
<position x="0.0" y="0.0"/>
<comment/>
<label>
<id>684e0d7d-0163-1000-c97e-dee22f4f649c</id>
<position x="268.0609696699687" y="-600.9202250695664"/>
<size height="761.6532745361328" width="473.57989501953125"/>
<id>684fefa4-0163-1000-542e-9343b5fa4f7f</id>
<position x="1145.4612992598124" y="-407.1201983696658"/>
<size height="170.00000762939453" width="416.8000030517578"/>
<styles>
<style name="font-size">14px</style>
<style name="font-size">12px</style>
</styles>
<value>NiFi-InfluxDB integration demo
=========================

This demo shows how InfluxDB can be integrated with the
NiFi data processing framework. To enable seamless
integration we implemented following NiFi extensions:

* InfluxDatabaseService, InfluxDatabaseService_2 – more NiFi
processors in flow can reuse one InfluxDB connection configuration
(url, credentials)
<value>Demo 2 User Story:

* PutInfluxDatabase, PutInfluxDatabase_2 – processors
that puts InfluxDB Line Protocol data into InfluxDB

As NiFi user I want to reliably monitor multiple environments
and be able to write data selectively.

* PutInfluxDatabaseRecord, PutInfluxDatabaseRecord_2 – processors
that puts structured data (json, csv, text, avro) into InfluxDB
using NiFi Record readers/writers.

* GetInfluxDatabase_2, GetInfluxDatabaseRecord_2 – processors that
are able use Flux query to get records from InfluxDB 2.0

* AC1: Partition and route metrics data using NiFi "Record Path language"
by type of measurement (name of container)
</value>
</label>
<label>
<id>280850da-016c-1000-bb1f-063422ae1933</id>
<position x="1149.274867625133" y="-17.059789260861407"/>
<size height="172.25083923339844" width="415.77381896972656"/>
<styles>
<style name="font-size">12px</style>
</styles>
<value>Demo 4 User Story:


* InfluxLineProtocolReader – reader parses the InfluxDB Line Protocol
into NiFi Record. This allows filtering, partitioning line protocol data in
NiFi flow produced by Telegraf.
As NiFi user I want to expose data from InfluxDB 2.0 on the particular
HTTP endpoint.

* InfluxLineProtocolRecordSetWriter – Writes the contents of a Record
Set as Line Protocol.</value>
* AC1: Exposing a result of static Flux Query as data for HTML widget
* AC2: Transforming Flux Result to XML or JSON - remote API
for server application</value>
</label>
<label>
<id>01631000-efa4-184f-e581-a743d2fe812a</id>
Expand Down Expand Up @@ -93,38 +91,40 @@ InfluxDB in order to work with time series.
</value>
</label>
<label>
<id>684fefa4-0163-1000-542e-9343b5fa4f7f</id>
<position x="1145.4612992598124" y="-407.1201983696658"/>
<size height="170.00000762939453" width="416.8000030517578"/>
<id>684e0d7d-0163-1000-c97e-dee22f4f649c</id>
<position x="268.0609696699687" y="-600.9202250695664"/>
<size height="761.6532745361328" width="473.57989501953125"/>
<styles>
<style name="font-size">12px</style>
<style name="font-size">14px</style>
</styles>
<value>Demo 2 User Story:
<value>NiFi-InfluxDB integration demo
=========================


As NiFi user I want to reliably monitor multiple environments
and be able to write data selectively.
This demo shows how InfluxDB can be integrated with the
NiFi data processing framework. To enable seamless
integration we implemented following NiFi extensions:

* AC1: Partition and route metrics data using NiFi "Record Path language"
by type of measurement (name of container)
</value>
</label>
<label>
<id>280850da-016c-1000-bb1f-063422ae1933</id>
<position x="1149.274867625133" y="-17.059789260861407"/>
<size height="172.25083923339844" width="415.77381896972656"/>
<styles>
<style name="font-size">12px</style>
</styles>
<value>Demo 4 User Story:

* InfluxDatabaseService, InfluxDatabaseService_2 – more NiFi
processors in flow can reuse one InfluxDB connection configuration
(url, credentials)

* PutInfluxDatabase, PutInfluxDatabase_2 – processors
that puts InfluxDB Line Protocol data into InfluxDB

* PutInfluxDatabaseRecord, PutInfluxDatabaseRecord_2 – processors
that puts structured data (json, csv, text, avro) into InfluxDB
using NiFi Record readers/writers.

* GetInfluxDatabase_2, GetInfluxDatabaseRecord_2 – processors that
are able use Flux query to get records from InfluxDB 2.0


As NiFi user I want to expose data from InfluxDB 2.0 on the particular
HTTP endpoint.
* InfluxLineProtocolReader – reader parses the InfluxDB Line Protocol
into NiFi Record. This allows filtering, partitioning line protocol data in
NiFi flow produced by Telegraf.

* AC1: Exposing a result of static Flux Query as data for HTML widget
* AC2: Transforming Flux Result to XML or JSON - remote API
for server application</value>
* InfluxLineProtocolRecordSetWriter – Writes the contents of a Record
Set as Line Protocol.</value>
</label>
<processGroup>
<id>2807fffd-016c-1000-ff4c-eb46f296840c</id>
Expand Down Expand Up @@ -1725,19 +1725,19 @@ curl -i -X GET -G http://localhost:8234 \
</property>
<property>
<name>Consumer Key</name>
<value>enc{6e527770c0199eaace82a11376faaeee802a18af8c8f9d8643693056850d00abde7ec6aafb7fb628fd7991e9657c5bc0}</value>
<value>enc{934a10f0c6c2cde7891c023473d0eff7c5b54a7f9b60a6571a1c106a5d5a671da9e2ea8b3c2e23abd79dd2821dc49a67}</value>
</property>
<property>
<name>Consumer Secret</name>
<value>enc{5f2a8265a805416e8dce0c90513ea94a8228803f9718f1c1bdc616c009e2408e14ab091df171f1454a14448b02f8b5807248a6e7e7e97add13b3eed5b7c0911adb446a8b2fa98a0214e4a1f0bbc6679a}</value>
<value>enc{cbbc9cf596e8e3081c7a01d5b14a18408a0397486f1f48ee778d536cbcfd68984f54d33a087746d004f747d78dc4bb0d3ae7107194e32185b1fac86d06d3c048bf1be5d9d5ef89ab30e7bab696d1bf0e}</value>
</property>
<property>
<name>Access Token</name>
<value>enc{a011c18126b15d0c3dd7387f5e66e75f11aa6b5a73bc6ed9a7853fc726f458d0352c95b421488f2c10dca4cabdc897d18b12fa0d76f9b1a9fa9e16e5216b9a67973f88272c379bcc3337547b578ea1cf}</value>
<value>enc{0acd71aa0121a10b60a853f26643b94c52e43aef08b235fe739ef098065b61b867ff49c8523e5dc22f62e011305dcaad11f8bf2728345d3e8deb08726aa0ab6855ff4a03fe9db70ba959103024968c61}</value>
</property>
<property>
<name>Access Token Secret</name>
<value>enc{29ff2b564fe3e7f5009fc56a3a72fbc2035803a3badff66d8b4cdeec858315a0a60bcf27d0ad8cf0844a0dc5cf6b84ff27359a3fed1e235b5db6abe94e1cf402}</value>
<value>enc{bb4bcb7a5225cee9ef41861400fe7204da4947f3757109b567b16a9f4af297639b8ed87faf52f55f657ae87ec70ece8d9f88ba00bb8610bea98c52c91ae8ee7e}</value>
</property>
<property>
<name>Languages</name>
Expand Down Expand Up @@ -2270,7 +2270,7 @@ curl -i -X GET -G http://localhost:8234 \
<processor>
<id>9993347a-5fbd-1c2b-a6cf-829e9b16c88d</id>
<name>PutInfluxDatabase</name>
<position x="1573.154113666188" y="683.2148985893302"/>
<position x="529.3148192325943" y="687.9812560112052"/>
<styles/>
<comment/>
<class>org.influxdata.nifi.processors.PutInfluxDatabase</class>
Expand Down Expand Up @@ -2331,7 +2331,7 @@ curl -i -X GET -G http://localhost:8234 \
<processor>
<id>dfceea1c-016b-1000-b2ce-2d172775840b</id>
<name>PutInfluxDatabase_2</name>
<position x="544.1536453794804" y="681.0541688763507"/>
<position x="1554.6283157896366" y="672.7129823529132"/>
<styles/>
<comment/>
<class>org.influxdata.nifi.processors.PutInfluxDatabase_2</class>
Expand Down Expand Up @@ -2464,8 +2464,8 @@ curl -i -X GET -G http://localhost:8234 \
<id>dfcf3302-016b-1000-50de-901254d75111</id>
<name/>
<bendPoints>
<bendPoint x="716.4171036314335" y="612.7235414349444"/>
<bendPoint x="664.8470352720585" y="608.0611268841632"/>
<bendPoint x="1825.7943619322148" y="605.5740053021319"/>
<bendPoint x="1743.2428482603398" y="599.7199403607257"/>
</bendPoints>
<labelIndex>1</labelIndex>
<zIndex>0</zIndex>
Expand Down Expand Up @@ -2591,8 +2591,8 @@ curl -i -X GET -G http://localhost:8234 \
<id>9993347e-5fbd-1c2b-e9aa-62ed6628b6f2</id>
<name/>
<bendPoints>
<bendPoint x="1734.5665282169693" y="618.2969603569084"/>
<bendPoint x="1775.805480853688" y="607.8701415092521"/>
<bendPoint x="690.7272337833756" y="623.0633177787834"/>
<bendPoint x="731.9661864200943" y="612.6364989311271"/>
</bendPoints>
<labelIndex>1</labelIndex>
<zIndex>0</zIndex>
Expand Down Expand Up @@ -3745,7 +3745,7 @@ curl -i -X GET -G http://localhost:8234 \
</property>
<property>
<name>influxdb-token</name>
<value>enc{c64de36e715cc2188839b440d41322345f888ba9013f2c5c259786c836a6cc59}</value>
<value>enc{b014836056e668bd988c8eeff7d219c00bfb7546131d1cb27aa87d7b634e384d}</value>
</property>
</controllerService>
<controllerService>
Expand Down
4 changes: 3 additions & 1 deletion scripts/nifi-restart.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -194,4 +194,6 @@ echo

curl -i -X POST -H "Content-Type: application/json" http://localhost:8888/chronograf/v1/dashboards -d @${SCRIPT_PATH}/chronograf/nifi-dashboard.json
curl -i -X POST -H "Content-Type: application/json" http://localhost:8888/chronograf/v1/dashboards -d @${SCRIPT_PATH}/chronograf/twitter-dashboard.json
curl -i -X POST -H "Content-Type: application/json" http://localhost:8888/chronograf/v1/dashboards -d @${SCRIPT_PATH}/chronograf/nifi-logs-dashboard.json
curl -i -X POST -H "Content-Type: application/json" http://localhost:8888/chronograf/v1/dashboards -d @${SCRIPT_PATH}/chronograf/nifi-logs-dashboard.json

#curl -i -X POST -H "Content-Type: application/json" -H "Authorization: Token my-password" http://localhost:9999/api/v2/dashboards -d @scripts/influx2/nifi_dashboard.json

0 comments on commit 3ce71f7

Please sign in to comment.