Add support placeholder for include directive

[y-ken]

Hi @repeatedly

How about to support hostname or short hostname placeholder in include directive?
This feature would bring us more flexible configuration management!

specification

• '/etc/td-agent' directory files are same in all servers.
• to select load file on startup fluentd, it uses hostname

use case

It is a way to switch behavior even though these are same configuration.

• [use case 1] fetch configuration from provisioning http server on startup.
• [use case 2] keep all server configuration as static file at '/etc/td-agent' directory from git repository.

sample configuration of use case 1

$ cat /etc/td-agent/td-agent.conf

include http://my-provisioning-server:8080/fluent/#{hostname}.conf

OR

$ cat /etc/td-agent/td-agent.rb

include {
  http://my-provisioning-server:8080/fluent/#{Socket.gethostname}.conf
}

sample configuration of use case 2

$ ls /etc/td-agent/
td-agent.conf
vms3021.conf
vms3022.conf
vms3023.conf
vms3024.conf
...snip...

$ cat /etc/td-agent/td-agent.conf

include /etc/td-agent/#{short_hostname}.conf

OR

$ cat /etc/td-agent/td-agent.rb

include {
  /etc/td-agent/#{Socket.gethostname.split('.')[0]}.conf
}

Thanks.

[repeatedly]

This feature seems to be realized by DSL configuration.

P.S.
In my personal opinion, hostname separation is not good for system management.
Hostname is sometimes changed so using hostname as filename is so error prone.
I think role based separation is good and there are popular tools which support role based management, e.g. Chef, Ansible and etc.

[y-ken]

I agree to your opinion.
At this time, it seems like DSL configuration does not support include directive.
Would you please tell me any example to load external configuration with DSL?

I have tried with this dsl. but it won't work because include method does not defined in lib/fluent/config/dsl.rb

$ cat /etc/td-agent/td-agent.rb

include("/etc/td-agent/${ENV['FLUENTD_ROLE']}.conf")

[repeatedly]

I'm not sure.

@tagomoris Do you hit a problem to support include feature in DSL?

[tagomoris]

It's just because lack of idea/time. External dsl configuration file can be included by require.

[y-ken]

Thank you for your comment, @repeatedly, @tagomoris .
Oh, I'll try it soon.

[y-ken]

@tagomoris
How do I load configuration with external uri?

[tagomoris]

It seems better to emulate @include of v1 configuration syntax, because configuration attribute name like @name is invalid.

There may be 2 options:

• Add instance variable @include, and set lambda to include external uri into it. For example in configuration file:

@include.("/path/to/configuration")

• Add instance variable @sys, @syntax or others such like these, and call include of this variable:

@sys.include "/path/to/configuration"

Latter one is better for the case which configuration files needs more @ keywords. Do you have any ideas about the number of keywords? @repeatedly

[repeatedly]

Another idea, source or match passes self to block.

source { |s|
  s.include "/path/to/configuration"
}

How about this? It reduces reserved variable.

[tagomoris]

@repeatedly How is it implemented on ROOT element?

[repeatedly]

@tagomoris Ah, I forgot this case. Hmm...
I don't use DSL configuration so I follow user's opinion.

[tagomoris]

Configuration DSL now supports include at master.