Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch telegraf data after disconnect leaves holes in continuous query tables #19698

Open
2clarkd opened this issue Oct 7, 2020 · 0 comments
Open

Comments

@2clarkd
Copy link

2clarkd commented Oct 7, 2020

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

  1. setup telegraf on host to send data
    [agent]
    interval = "60s"
    round_interval = true
    metric_batch_size = 1000 // allow batched data
    metric_buffer_limit = 10000 // allow sufficient data aggregation over disconnect period
    ...

  2. setup second host with influx and continuous query
    DROP CONTINUOUS QUERY "cq.starts" ON "app";
    CREATE CONTINUOUS QUERY "cq.starts" ON "app"
    BEGIN
    SELECT
    non_negative_difference(max(global_starts))
    as starts,
    INTO "app"."one-year"."summary.starts"
    FROM "app"."autogen"."data.starts"
    GROUP BY time(1m), * fill(none)
    END;

  3. take down host running influx for a period of time (say 15minutes) and restart

  4. view graphs using the continuous query table and note 15minute gap of data

Expected behavior:
Describe what you expected to happen.
The continuous query should process the delayed batch data and fill in the hole created during the disconnected interval when the system is re-connected.

Actual behavior:
Describe What actually happened.
The continuous query data is left blank for the interval.

Environment info:

  • System info: Run `` and copy the output here
    inux influxdb-5786ccb9f8-pjmqj 3.10.0-862.el7.x86_64 Queries should support an order clause and limit clause #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 Linux
  • InfluxDB version: Run influxd version and copy the output here
    InfluxDB v1.7.8 (git: 1.7 ff383cd)
  • Other relevant environment details: Container runtime, disk info, etc

Config:
Copy any non-default config values here or attach the full config as a gist or file.

Logs:
Include snippet of errors in log.

Performance:
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

# Commands should be run when the bug is actively happening.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant