Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: upgrade Flux to v0.83.2 #19569

Merged
merged 17 commits into from
Sep 17, 2020
Merged

feat: upgrade Flux to v0.83.2 #19569

merged 17 commits into from
Sep 17, 2020

Conversation

wolffcm
Copy link

@wolffcm wolffcm commented Sep 16, 2020

Closes #19468

This upgrades Flux to version 0.83.2.

In the first commit I just copied the old (before the split) contents of influxdb/query/ into the current master. So, you may find it helpful to look at this diff, that compares that commit to the head of this branch:
2438776...feat/upgrade-flux-to-v0.83.1

This makes it easier to see what I changed by hand.

Here's a summary of what I did:

  • Returned to the modern way of initializing Flux runtime
  • Restored cmd/influxd/launcher/query_test.go to its previous state (still skipping pushdown tests)
  • Restored http/query.go and http/query_test.go to their previous state
  • Reapplied many, but not all, of the commits from this issue: Port rejected commits after Flux has been upgraded #19444 (more on this below)
  • Removed temporary copy of modern Flux in pkg/flux, and updated references to it
  • In query/ most of the code is the same as before the split, except the code for to() and experimental.to(). Here I left in @stuartcarnie 's changes that make these both work.

On the commits from #19444 that didn't get reapplied after the split, I kept all of them except:

  • revert(storage): Revert feat(influxql): Implement DELETE & DROP MEASUREMENT 1501351 cc/ @benbjohnson
    That one seems like it will need to be ported to work with the new storage engine?

I also kept the commits that add a new feature flag for injecting task last success time, since it reapplied cleanly:

There were several Flux end-to-end-tests that failed after upgrading Flux. I filed issues for them:

@wolffcm wolffcm requested a review from a team September 16, 2020 02:02
@wolffcm wolffcm requested a review from a team as a code owner September 16, 2020 02:02
@wolffcm wolffcm force-pushed the feat/upgrade-flux-to-v0.83.1 branch from 7b1009d to c8cbd41 Compare September 16, 2020 02:14
Copy link
Member

@jacobmarble jacobmarble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The small quantity of storage code LGTM

@stuartcarnie
Copy link
Contributor

@wolffcm / @benbjohnson I re-implemented DELETE / DROP MEASUREMENT via #19549 to talk to TSM1

@wolffcm wolffcm force-pushed the feat/upgrade-flux-to-v0.83.1 branch from 5a8a043 to c711c40 Compare September 16, 2020 17:01
Copy link
Contributor

@stuartcarnie stuartcarnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me – great work, @wolffcm!

I ran this branch locally, feeding it some telegraf data and verified the output of Flux and InfluxQL. The only difference is where the windows start, but otherwise the numbers match 💯

InfluxQL

Query

curl -v -u ${INFLUX_USER}:${INFLUX_TOKEN} -H Content-Type:application/vnd.influxql -H Accept:text/plain ${INFLUX_HOST}/query\?db=my-bucket --data-binary "select mean(usage_system), mean(usage_user) from cpu where cpu='cpu-total' and time > now() - 1h group by time(1m) fill(none)"

Output

name: cpu
time                          mean               mean_1
----                          ----               ------
2020-09-16 19:41:00 +0000 UTC 2.9198681258211403 5.292570832522467
2020-09-16 19:42:00 +0000 UTC 1.9832891660092988 3.058275498670035
2020-09-16 19:43:00 +0000 UTC 2.544792363245724  4.052888237566831
2020-09-16 19:44:00 +0000 UTC 1.8886189464117715 3.1189570386709
2020-09-16 19:45:00 +0000 UTC 1.841633520255214  2.568726119485858
2020-09-16 19:46:00 +0000 UTC 1.7749434246601743 3.0915269274326174
2020-09-16 19:47:00 +0000 UTC 1.812600693052109  3.427265186672049
2020-09-16 19:48:00 +0000 UTC 1.7500492149990279 2.8813245783993326
2020-09-16 19:49:00 +0000 UTC 1.9603237111153404 3.361274240750055
2020-09-16 19:50:00 +0000 UTC 1.682340091326152  2.502154705310155
2020-09-16 19:51:00 +0000 UTC 1.80315887143924   2.8188620228994257
2020-09-16 19:52:00 +0000 UTC 1.7530775060454051 3.534266400071696
2020-09-16 19:53:00 +0000 UTC 2.06786301770364   4.019988594516871
2020-09-16 19:54:00 +0000 UTC 1.8425985358743213 3.747834838625534
2020-09-16 19:55:00 +0000 UTC 1.6385691186518805 3.852273992401868
2020-09-16 19:56:00 +0000 UTC 1.5833621177356518 3.4416504741237843
2020-09-16 19:57:00 +0000 UTC 1.6311448927276098 4.307079769121812
2020-09-16 19:58:00 +0000 UTC 1.5969358803294724 3.0137355054165647
2020-09-16 19:59:00 +0000 UTC 1.6029566972286364 2.012266481482333

Flux

Query

curl -v -H 'content-type:application/vnd.flux' -H "Authorization: Token ${INFLUX_TOKEN}" ${INFLUX_HOST}/api/v2/query\?org\=my-org --data-binary 'from(bucket: "my-bucket") |> range(start: -1h) |> filter(fn: (r) => r["_measurement"] == "cpu" and (r["_field"] == "usage_user" or r["_field"] == "usage_system") and r["cpu"] == "cpu-total") |> aggregateWindow(every: 10s, fn: mean, createEmpty: false) |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value") |> drop(columns:["_start", "_stop", "host", "_measurement", "cpu"])'

Output

,result,table,_time,usage_system,usage_user
,_result,0,2020-09-16T19:42:00Z,2.9198681258211403,5.292570832522467
,_result,0,2020-09-16T19:43:00Z,1.9832891660092988,3.058275498670035
,_result,0,2020-09-16T19:44:00Z,2.544792363245724,4.052888237566831
,_result,0,2020-09-16T19:45:00Z,1.8886189464117715,3.1189570386709
,_result,0,2020-09-16T19:46:00Z,1.841633520255214,2.568726119485858
,_result,0,2020-09-16T19:47:00Z,1.7749434246601743,3.0915269274326174
,_result,0,2020-09-16T19:48:00Z,1.812600693052109,3.427265186672049
,_result,0,2020-09-16T19:49:00Z,1.7500492149990279,2.8813245783993326
,_result,0,2020-09-16T19:50:00Z,1.9603237111153404,3.361274240750055
,_result,0,2020-09-16T19:51:00Z,1.682340091326152,2.502154705310155
,_result,0,2020-09-16T19:52:00Z,1.80315887143924,2.8188620228994257
,_result,0,2020-09-16T19:53:00Z,1.7530775060454051,3.534266400071696
,_result,0,2020-09-16T19:54:00Z,2.06786301770364,4.019988594516871
,_result,0,2020-09-16T19:55:00Z,1.8425985358743213,3.747834838625534
,_result,0,2020-09-16T19:56:00Z,1.6385691186518805,3.852273992401868
,_result,0,2020-09-16T19:57:00Z,1.5833621177356518,3.4416504741237843
,_result,0,2020-09-16T19:58:00Z,1.6311448927276098,4.307079769121812
,_result,0,2020-09-16T19:59:00Z,1.5969358803294724,3.0137355054165647
,_result,0,2020-09-16T19:59:33.855846Z,1.6604894184520498,2.1355091120673584

@stuartcarnie
Copy link
Contributor

👏 👏

@wolffcm wolffcm merged commit e7cbbaa into master Sep 17, 2020
@wolffcm wolffcm deleted the feat/upgrade-flux-to-v0.83.1 branch September 17, 2020 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update Flux version to v0.83.1
4 participants