Influxdb storage for dashboard leads to generate a lot of shard in influxdb in the past

[lpapier]

Hi,
I'm running grafana 1.7.0rc1 with influxdb 0.8.0rc4.
After storing my dashboard into my influxdb, I started to see influxdb generating a new shard in the past every 10 minutes. My guess is that it is related to the time provided to store the dashboard.
In my case 1000000000000000u (Sunday 09 Sep 2001, 03:46:40). All my system clock are in sync with ntp servers.

Here is the log of the influxdb dashboard creation:

[2014/08/07 15:19:18 CEST] [INFO] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).RunQuery:81) Start Query: db: grafana, u: grafana, q: select title, tags from /grafana.dashboard_.*/ where  title =~ /.*.*/i
[2014/08/07 15:20:08 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:842) No matching shards for write at time 1000000000000000u, creating...
[2014/08/07 15:20:08 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:873) createShards for space default: start: Thu Sep 6 02:00:00 +0200 CEST 2001. end: Thu Sep 13 02:00:00 +0200 CEST 2001
[2014/08/07 15:20:08 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:163) DATASTORE: opening or creating shard /home/influxdb/db/shard_db_v2/00002
[2014/08/07 15:20:08 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1090) Adding shard to default: 2 - start: Thu Sep 6 02:00:00 +0200 CEST 2001 (999734400). end: Thu Sep 13 02:00:00 +0200 CEST 2001 (1000339200). isLocal: true. servers: [1]
[2014/08/07 15:20:08 CEST] [INFO] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).RunQuery:81) Start Query: db: grafana, u: grafana, q: select dashboard from "grafana.dashboard_RG9tb3RpcXVlIFBlcnNv"

Here are some log of shard creating in the past:

[2014/08/07 15:22:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).automaticallyCreateFutureShard:233) Automatically creating shard for Thu Sep 13 02:00:01 +0200 CEST 2001
[2014/08/07 15:22:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:873) createShards for space default: start: Thu Sep 13 02:00:00 +0200 CEST 2001. end: Thu Sep 20 02:00:00 +0200 CEST 2001
[2014/08/07 15:32:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).automaticallyCreateFutureShard:233) Automatically creating shard for Thu Sep 20 02:00:01 +0200 CEST 2001
[2014/08/07 15:32:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:873) createShards for space default: start: Thu Sep 20 02:00:00 +0200 CEST 2001. end: Thu Sep 27 02:00:00 +0200 CEST 2001
[2014/08/07 15:42:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).automaticallyCreateFutureShard:233) Automatically creating shard for Thu Sep 27 02:00:01 +0200 CEST 2001
[2014/08/07 15:42:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:873) createShards for space default: start: Thu Sep 27 02:00:00 +0200 CEST 2001. end: Thu Oct 4 02:00:00 +0200 CEST 2001
[2014/08/07 15:52:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).automaticallyCreateFutureShard:233) Automatically creating shard for Thu Oct 4 02:00:01 +0200 CEST 2001
[2014/08/07 15:52:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:873) createShards for space default: start: Thu Oct 4 02:00:00 +0200 CEST 2001. end: Thu Oct 11 02:00:00 +0200 CEST 2001
[2014/08/07 16:02:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).automaticallyCreateFutureShard:233) Automatically creating shard for Thu Oct 11 02:00:01 +0200 CEST 2001
[2014/08/07 16:02:00 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:873) createShards for space default: start: Thu Oct 11 02:00:00 +0200 CEST 2001. end: Thu Oct 18 02:00:00 +0200 CEST 2001

[torkelo]

Why does it generate a new shard? I use 1000000000000000 as fixed time to save dashboards to (so I can overwrite them). When I write the same timestamp and sequence nr why create new shards all the time?

[torkelo]

ping @pauldix

[lpapier]

Hi,
I was not precise enough in my report. The 'automaticallyCreateFutureShard' are not directly triggered by grafana write activity.

My guess is that writing a data point in the past triggers some internal routine in influxdb. It's like influxdb fill the gap between the older shard and the newer one.

@pauldix could certainly clarify this point.

[pauldix]

In Influx 0.7, once you create a shard in either the short term or long term space, it will then create shards for time blocks going forward from now. However, it shouldn't create a bunch of shards in between.

This could get a little tricky with 0.8 because people can define their own shard spaces and retention policies. So They'd need to make sure that there's a shard space that keeps data around forever that the Grafana dashboards should get written to.

What you're seeing in the logs is odd. I'll have to look into it. Influx shouldn't be creating shards for the time periods between the dashboard time and current time.

[tmonk42]

I saw the same behavior.

[lpapier]

@pauldix I'm using the default shard configuration from influxdb 0.8.0rc4.

I known that Grafana dashboard will be removed after 7 days. I'm still testing influxdb/grafana, so it is not an issue for the moment.

[michail-nikolaev]

any news/ideas?

[torkelo]

@michail-nikolaev no, I am still unsure how reliably this will be. Worried that people will loose dashboards because of shard retention policies.

[jordanrinke]

with 0.8 and the ability to define storage periods it makes it a non issue. Just in the tutorial for setup require a share space be infinte for the matching dashboard storage name.

[lpapier]

@jordanrinke I'm using influxdb 0.8.0rc4 with the default config. The default shard config seems to be infinite retention. So yes, with the default shard config in 0.8 grafana dashboard will not be erased by influxdb shard management. But this is not the issue I have reported.

The issue is that influxdb generate a lot a (empty?) shards to fill the gap between 1000000000000000 timestamp and the newest shard. With the default config one shard per 7 days...

I agree that the issue/bug is not in grafana. But I'm sure that grafana/influxdb users don't want to have all those shards created in their influxdb.

[huhongbo]

any update? I want to switch to influxdb for dashboard storage
if the problem exists I should wait

[torkelo]

@huhongbo maybe ask in #influxdb on freenode or open an issue in the influxdb repo (if there is non already)

[huhongbo]

@torkelo OK , or maybe change the 1000000000000000 to current ,bcs the 0.8 influxdb default set inf on the shard

[torkelo]

I can't change to current, It needs to fixed, or else each save will create a new dashboard. I could change the fixed value to be more current date, but will only push the problem into the future.

[rishid]

I am also seeing this, Influx DB has created 118 shards of ~9 MB each just for the Grafana database, which currently has 1 data point in it.

Why not just use the current time and query the latest copy of the dashboard using sorting and LIMIT 1?

[toddboom]

@torkelo Maybe creating new dashboards (and thus keeping around old ones) is a better solution than writing to (and updating) an arbitrary timestamp. Then you could just pull back the most recent dashboard with a LIMIT 1.

I can also imagine that some simple method of culling old dashboards could be concocted as well.

[torkelo]

How would search work then? So you only get the latest versions

[huhongbo]

I think this problem can be avoid by setting the shard during to 365d or more large

[rishid]

Good idea @huhongbo

I don't know the exact scheme grafana is storing dashboards but if you make a series per dashboard

A query as simple as this will give you the most recent point for every series:

select * from /.*/

[jvshahid]

This was fixed on master, see influxdata/influxdb#954 for more discussion and workarounds.

[jvshahid]

Just to be more clear, InfluxDB won't create more shards than necessary when grafana stores the dashboard using an old timestamp.