datasource config support for v2

[brettswift]

According to #35 it's possible to configure datasources via the API, but the work hasn't been done yet.

I didn't see anywhere to track this work - so I thought I'd create this issue card. I may / may not have time eventually to work on this, however not for the next month or so.

Let me know if this is being tracked elsewhere, or if anyone is working on this feature.

Thanks!

[bfraser]

Hi @brettswift thanks for your interest! This is not being tracked elsewhere, nor is anyone working on the feature (that I am aware of).

I don't know how much of a use case there is for managing Grafana's data sources as resources in Puppet, since it tends to be something you configure initially and then leave alone, and they can easily be managed via the web interface.

That being said, if there is a desire for such functionality in the module, and you have time available to work on it, I would appreciate contributions and would assist in review / testing.

Documentation for working with Grafana's data sources using its HTTP API is available here.

[simonpasquier]

Hi, I've got this working in another Puppet module [0] for my own project. I'm willing to contribute it to this module if others are interested. Just need to find a bit of time to send a PR.

[0] https://github.com/openstack/fuel-plugin-influxdb-grafana/blob/master/deployment_scripts/puppet/modules/lma_monitoring_analytics/lib/puppet/type/grafana_datasource.rb
https://github.com/openstack/fuel-plugin-influxdb-grafana/blob/master/deployment_scripts/puppet/modules/lma_monitoring_analytics/lib/puppet/provider/grafana_datasource/grafana.rb

[brettswift]

The use case for us would be kind of useful so I'll try to explain. I think it's likely useful for others but that's just my opinion :)

Without the ability to configure this you're forced to use your mouse and click around, and copy and paste. Which isn't great. We try to have everything driven by configuration data, which helps us automate.

In a long lived server, this isn't that beneficial, but the argument is that even once it's useful because I don't have to give instructions to our operations team.

Where it's really useful is in ephemeral test environments where people are buliding isolated stacks to test with, and where they want to test thier monitoring alerts and checks. We could give an app stack to 20 different people, each with an isolated monitoring stack as well. They may use this for a couple of days and throw it away.. having to go in everytime and add data sources is a pain where we could have had it automatically configured.

Hopefully that helps - 2 use cases.. one for developing and one for production or through dev->test-> staging-> production control. (ie: with a new release there is a new configuration going up, that includes a new datasource and should only be added at release time).

Does that help?

[ltutar]

I am using Puppet to roll out a continuous delivery pipeline. I would like to configure the datasource through this module in order to integrate elasticsearch with grafana. It is then fully automated and all of the functionality is in one place i.e. this module.

[simonpasquier]

I've sent a PR with what I had already in my other module. Note that it only supports InfluxDB because that's the backend we're using but I think that additional backends could be supported easily.

Let me know if this matches your requirements.

[ltutar]

Definitely if I can define the elasticsearch as type and the hash properties through yaml.

[simonpasquier]

@ltutar I've updated my PR to support more datasource types. Can you have a look and try? I'm primarily an InfluxDB user...

[ltutar]

I'll give it a try this weekend.

[simonpasquier]

@ltutar any feedback on my patch?

[ltutar]

It works like a charm. Thank you.
My code:

# == Class grafana_config::config
#
# This class is called from grafana_config for service config.
#
class grafana_config::config {

  grafana_datasource { 'elasticsarch':
    grafana_url       => 'http://192.168.234.73:3000',
                      grafana_user      => 'admin', 
                      grafana_password  => 'admin', 
                      type              => 'elasticsearch',
                      url               => 'http://localhost:9200',
                      access_mode       => 'proxy', 
                      is_default        => true,    
                      database          => 'logstash-*',
                      json_data         => {        
                        "timeField" => "@timestamp",
                        "esVersion" => 1
                      }           
  }     
}

The next step is to yaml the code :)

[ltutar]

Puppet agent run does each time notices the json_data change. I do not know what the implication of this action is.

[root@elkagent ~]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for elkagent.home
Info: Applying configuration version '1451998051'
Notice: /Stage[main]/Grafana_config::Config/Grafana_datasource[elasticsarch]/json_data: json_data changed '' to '{"timeField"=>"@timestamp", "esVersion"=>1}'
Notice: Applied catalog in 16.30 seconds
[root@elkagent ~]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for elkagent.home
Info: Applying configuration version '1451998079'
Notice: /Stage[main]/Grafana_config::Config/Grafana_datasource[elasticsarch]/json_data: json_data changed '' to '{"timeField"=>"@timestamp", "esVersion"=>1}'
Notice: Applied catalog in 11.04 seconds
[root@elkagent ~]# 

[simonpasquier]

@ltutar I'll have a look at it.

[ltutar]

Also, with the first puppet run, the datasource is called before the grafana install. the second run is OK.

Error: /Stage[main]/Grafana_config::Config/Grafana_datasource[elasticsarch]: Could not evaluate: Connection refused - connect(2) for "localhost" port 3000

[simonpasquier]

@ltutar I guess you need to set an explicit dependency between the grafana service and the grafrana_dashboard resource(s).

[ltutar]

for future reference of dependency issue:
init.pp

  class { '::grafana_config::config': 
     require => Service['grafana-server']
  } 

config.pp

class grafana_config::config {

  exec {'wait_for_grafana':
    command => '/usr/bin/wget --tries 10 --retry-connrefused --no-check-certificate http://localhost:3000',
  }

  grafana_datasource { 'elasticsarch':
    grafana_url       => 'http://localhost:3000',
    grafana_user      => 'admin',
    grafana_password  => 'admin',
    type              => 'elasticsearch',
      url               => 'http://localhost:9200',
      access_mode       => 'proxy',
      is_default        => true,
      database          => 'logstash-*',
      json_data         => {
        "timeField" => "@timestamp",
        "esVersion" => 1
    }
  }

  Exec['wait_for_grafana'] ->
  Grafana_datasource['elasticsarch']
}

[simonpasquier]

I've noticed a bug in Grafana that explains why the resource tries to always update the json_data field.

[ltutar]

thank you

[simonpasquier]

So I've updated the PR to take into account how the Grafana API works. The CI tests are failing but I'm not sure it's related to my change since I see that many other PRs are in the same state.

[simonpasquier]

I've got another PR that somehow fixes the CI issue.

[nvtkaszpir]

looks like grafana_datasource requires auth.basic to be enabled?

[simonpasquier]

@nvtkaszpir yes, feel free to open an issue if you want it to support other methods.