Discussion around using variables in Caddyfile

[brucejo75]

I was reading the variable assignment thread in the forum essentially outlining that you could assign a placeholder using map or import. The author of the post specifically asked about being able to assign non-environmental variables (e.g. placeholders).

It occurred to me that "Environment Variables" could be more generalized to "Variables" if you require a declaration before usage...

I would like to do something like this:

:55555 {
{$MY_BASE_SITE=env.MYBASESITE | mysite.com}
  map {host} {port} {
    "a.{$MY_BASE_SITE}" 1111
    "b.{$MY_BASE_SITE}" 2222
  }

  reverse_proxy 0.0.0.0:{port}
}

I can make this work by assigning an environment variable before running caddy, but oftentimes it would be useful to declare in the Caddyfile (e.g. use of more than a few variables).

Feature Request

• Create a new variable type, modelled after Environment Variables with the requirement that they need to be explicitly declared.

# Always need to declare variables before using them
# set to DEFAULTVALUE unless env.VARIABLE exists
{$VARIABLE=env.VARIABLE | DEFAULTVALUE}

# subsequent references will be equal to whatever was assigned e.g. will be DEFAULTVALUE
# if env.VARIABLE is not defined
respond {$VARIABLE}

# and a default override in place could still be used in the reference
respond {$VARIABLE:foobar}

# if env.VARIABLE2 does not exist VARIABLE2 will be empty
{$VARIABLE2= env.VARIABLE2}

#invalid because there is no declaration e.g. {$VARIABLE3=...}
{$VARIABLE3}

[mholt]

(Your original post in the website repo was the start of a great wiki entry you could submit!)

Variables can already be declared with the vars HTTP handler module.

(This is how the root directive works, it just sets a var called "root".)

Other than the map directive, I'm not sure we directly expose the vars module to the Caddyfile (with map being a fancy way to assign variables based on conditions, and vars would just be a direct assignment).

[brucejo75]

Thanks @mholt! This makes me laugh, I will drag you through my process... (I am sharing it because, most of my attempts to learn something new about caddy go like this)

I read vars HTTP handler module.

---

Thought bubble:

Hmm, what is the mystical block that has 2 empty strings in it? a typo? Which part is a typo? Maybe an omission? Maybe there is some context somewhere... Why isn't there an example?
Maybe there is more info in the vars documentation? No help there. That is just a matcher.
Let me try guessing...

Try 1

Is it a global options block?

{
    "hello":"foobar"
}

:2015 {
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:4: unrecognized global option: Hello

Try 2

Maybe it should be a vars block?

vars {
    "hello":"foobar"
}

:2015 {
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:4: unrecognized directive: Hello

Try 3

Maybe put it in the Site block?

:2015 {
{
  "hello":"foobar"
}
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:7 - Error during parsing: Site addresses cannot contain a comma ',': '{https.vars.hello}, world! {https.vars.hello}' - put a space after the comma to separate site addresses

Try 4

Maybe I need to put it in a vars block inside the global block?

{
  vars {
    "hello":"foobar"
  }
}

:2015 {
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:3: unrecognized global option: vars

Try 5

I know maybe it is a vars block in the Site block!

:2015 {
{
  "hello":"foobar"
}
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

grrr: run: adapting config using caddyfile: CaddyTest:3: unrecognized global option: vars

Oh yeah, I get most questions answered by going through the forum.

errr, search for "vars" turns up some references but I cannot find vars on most of the pages.

Does the module need to be installed?

The module claims it is standard so it should be in the executable... No luck there.
Look at the go code? I see the SetVar but that is no help.

How do I reference a module? I cannot find anything on it in the docs. I included the tls godaddy module and that extends the tls directive but the documentation for the godaddy module is just as opaque. Luckily, it was all explained in the github.

Wait look for http.vars on the forum!

Found this, a clue!

Wait a minute, those are JSON files. What did @mholt say above?

I'm not sure we directly expose the vars module to the Caddyfile (with map being a fancy way to assign variables based on conditions, and vars would just be a direct assignment).

D'oh! This is a JSON only feature? let me try it

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":2015"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "vars",
                  "hello": "foobar"
                },
                {
                  "body": "{http.vars.hello}, world! {http.vars.hello}",
                  "handler": "static_response"
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Works like a charm! I guess it is not available in a Caddyfile, that's a shame 😞

Observations

• The module documentation is woefully lacking.

{
 "":""
}

What the heck is that?

• Maybe good to add a JSON example and a Caddyfile example.
• If it does not work in Caddyfile, say so.
• Maybe some more info on how modules integrate with caddy. Apparently, the handler keyword is important. I do not see that described in the docs? At least when I search for the word "handler" the only real hit is this.

erg: 2 hours I will never get back

[mholt]

@brucejo75 Well, thank you for writing all that out. 😅 A very insightful read.

A few thoughts:

As you discovered, these are the JSON config docs. Caddy's native configuration is JSON, and we emphasize this from the Getting Started page. In other words, this is not the area of the site describing the Caddyfile config. I could have been clearer in my reply above, where I tried to contrast that with the Caddyfile (when I said "I'm not sure we directly expose the vars module to the Caddyfile"), but upon re-reading it, I realize why that wasn't clear and I'm sorry for the confusion. I was just saying that there is a module that lets you define variables. It just has to be done with JSON. Also sorry for the lost time.

I think the module documentation is very clear and concise:

The key is the variable name, and the value is the value of the variable. Both the name and value may use or contain placeholders.

Hence, the two empty strings: { "": "" }. It's a key and value of a JSON object. And the docs say exactly what they are.

We could probably use examples; it's always hard to draw the line between reference documentation and examples though. I like keeping them separate to maintain good organization.

For examples, we have our wiki but not nearly enough contributors. ☹️ Actually, your discoveries about the vars module would be a handy addition to the wiki. You should post there!

I guess it is not available in a Caddyfile, that's a shame 😞

No one has asked for it before now. 🤷‍♂️ I guess we could go ahead and add it. Any thoughts @francislavoie ?

Again, I really appreciate your time here, and apologize for the confusion. This is just a feature that not many people have used or asked about.

[francislavoie]

Yeah we should add the Caddyfile directive, it was an oversight. No harm in it existing.

It's really not that useful on its own though, IMO. map is much more flexible.

[mholt]

Real quick, before we do add a vars directive, is it already possible to simply "set a variable" using the map directive?

[francislavoie]

Yes but it's always a 3-liner and it's kinda awkward I guess:

map 1 {output} {
    1 value
}

Something like that might work. But obviously it will still cause a string comparison on every request.

[mholt]

Ah, clever. That is kinda awkward, you're right.

What about a special-case syntax for the map directive that basically accepts map <var> <value> (no block)? As a shorthand. That way, you just use map to assign variables, period.

[francislavoie]

I'm not a big fan of overloading directives that much. It's cheap to just add vars as a directive IMO

[mholt]

It's cheap, but IMO it makes more sense to have map be the directive that sets variables. 🤷‍♂️

[brucejo75]

Am I correct in assuming that the vars directive only sets placeholders? If so, the title: "Expose 'vars' handler as Caddyfile directive" is not what I was requesting or need. I am requesting an extension of how environment variables work.

I am requesting variables that get substituted before parsing not during runtime.

1. setting a variable via map works and I am OK with it if that is the way to go for placeholders. No need to change anything for setting placeholders, there is a reasonable workaround via map.
2. I could not accomplish what I want to do via placeholders in my original example. I tried setting map parameters with placeholders and it does not work. And it does not make sense for it to work. The map parameters are defining a fixed table at parse time I believe. That is why I really want more out of the environment variable paradigm.

But a real objection is the documentation. Guys, this stuff is hard to connect the dots on... (And it is a shame, because the product is EXCELLENT!)
As an example, here are some small tweaks to the vars module documentation, that would have saved me hours:

Change the top portion to lay out the entire JSON object:

JSON Directive:
{
      "handler" "vars",
      "<keyword>": "<value>
}

Caddyfile Directive:
-NOT AVAILABLE-
Note: placeholders can be set using the `map` directive in Caddyfiles.

These simple changes to be more explicit would have saved me 2 hours. I am happy to take a shot at fixing these things as I run across them and submitting PRs.

In my limited experience with caddy you have made very reasonable engineering tradeoffs (you are good at what you do). But discovery of these tradeoffs is an exercise in scanning everywhere docs (in multiple places) & the forum). I just think it would be easy to spell the tradeoffs out explicitly in the docs.

[francislavoie]

Am I correct in assuming that the vars directive only sets placeholders?

Doesn't exactly set placeholders, it sets "vars" in the request context, which can be used with a placeholder (slight distinction).

The map directive also does that but it's basically a fancy switch statement to set one or more "vars" at the same time.

I am requesting variables that get substituted before parsing not during runtime.

That's exactly how they work right now, in the Caddyfile {$ENV} style, they get replaced at Caddyfile adapt time.

Defining new variables though, would be a lot of added complexity I think, and I don't think that's warranted here. Just set your variables ahead of time.

Or use snippets to reuse some common bits of config https://caddyserver.com/docs/caddyfile/concepts#snippets

As an example, here are some small tweaks to the vars module documentation, that would have saved me hours

The JSON docs are auto-generated from the code's comments and Go code structure. With the way it's done right now, there's currently no way to control what those string values end up containing in the generated output.

But noted, that could be a future addition to the JSON doc generator, but it would also require adding something to the code to fill that in.

Caddyfile Directive:
-NOT AVAILABLE-
Note: placeholders can be set using the map directive in Caddyfiles.

I don't think this makes sense. If you're in the JSON docs, you shouldn't assume at all that any of it is available in the Caddyfile.

The Caddyfile adapter outputs a JSON config, but JSON can't go back to Caddyfile. It's a one-way relationship.

[brucejo75]

This thread now has 3 separate concepts:

vars does not accomplish what I am looking for lets stop talking about it

Example:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":2015"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "vars",
                  "base": "mysite.com"
                },
                {
                  "defaults": [
                    "a.{base}"
                  ],
                  "destinations": [
                    "{myvar}"
                  ],
                  "handler": "map",
                  "source": "1"
                },
                {
                  "body": "myvar: {myvar} base: {http.vars.base}",
                  "handler": "static_response"
                }
              ]
            }
          ]
        }
      }
    }
  }
}

curl localhost:2015: myvar: a.{base} base: mysite.com
lets take vars out of the conversation

Environment variables almost do it (for Caddyfiles only)

All that I am saying is "Environment variables are great" and can be used to accomplish what I need. For cases where there are only 1, 2 or 3 variables Environment variables do the job as is.

For cases where there might be a lot more parameters in a file it would be useful to:

• Be able to initialize environment variables within the Caddyfile rather than their value always coming in from the environment.

Your users are programmers we want to parameterize our config files.

This seems very intuitive to me that a developer would want to make their config file parameterized. This is a requested feature of nginx, but nginx is resisting the feature. One user even developed his own pre-processor that he has shared: https://stackoverflow.com/a/15432888/7669642 (this was 9 years ago, but people are still responding to it as recently as 2020)

Possible reactions to this request:

• This is a great idea and would differentiate us from nginx, we should think about how we could support macro expansion within the config files (and within json files!). Stay tuned it may take a while.
• We are better than nginx that's good enough.
• Ugh, too much work. We have other more important things to do. Maybe a later date.
• This an unreasonable idea, nobody would want this. Don't think anyone would want more than 2-3 variables in a config file.
• Hey have you tried vars? (yes, they do not accomplish the task)
• Have you tried snippets? (yes, I have used snippets to great effect but I cannot use them to parameterize the map parameters)

Documentation has issues

I will open separate issues in https://github.com/caddyserver/website

[mholt]

Ah, I see. You are requesting a programmable Caddyfile.

I will have to read that link when I am not mobile, but I also resist this notion, instinctively. There is some history here... when I was designing Caddy 2 I contemplated and experimented and debated for months whether the config should be declarative or imperative. We obviously settled on declarative and there are a lot of reasons for this -- pros and cons for sure -- but mostly pros.

Now, it turns out you can get in trouble mixing imperative and declarative.

Ultimately, it boils down to you being able to start with a parameterized document and evaluate it into a declarative document, but you can't go the other way around, and you have to be very careful when combining them.

So, in other words, the Caddyfile (or ideally, JSON) document you give Caddy should be the output of your parameterization, not the input.

I may respond to this more once I get back into the office, but I wanted to dump a few initial thoughts for now.

I think there is some good discussion here and even if it's not the point, I now want to enhance the map directive, improve documentation, and consider other elements you have brought up.

(So I'm leaning toward the first of your "possible responses" but I need to evaluate things more.)

[brucejo75]

@mholt , @francislavoie, following up on @mholt's comment. I took some of my learnings and put together a gist: Variables in Caddyfiles

If you would not mind examining for bonehead errors or glaring omissions I would appreciate it, I am still pretty green in the world of Caddyfile authoring.

I will post on the Wiki after your review.

Thanks!

[mholt]

@brucejo75 Great, I left some comments there. Am also drafting up a new commit to enhance the map handler a little more to support simple variable setting (without having to use the 1 hack) and to support using placeholders in the table.

[mholt]

I actually ended up adding the vars directive to simply set variables without conditions/switches (@francislavoie should be pleased). See 79cbe7b.

[brucejo75]

Thoughts/reactions from a green user...:

Will having a vars directive and a vars request matcher be confusing? To be fully literal the matcher is really isVarOneOf. I wonder if there is a better name for the matcher?

Can more than one variable be set with the vars directive? If not, I wonder if the directive shouldn't be named var?

[francislavoie]

@brucejo75 I don't think so, we have the try_files directive and the try_files matcher option for the file matcher, etc. The docs links things together pretty well, you can click directive tokens to go to the directive page, or matcher tokens to go to the matcher in question on the matchers page.

I think it's valid to say the directive should be var instead of vars though, we don't call the existing one roots or maps 🤔

[mholt]

@brucejo75

Can more than one variable be set with the vars directive?

Yes:

vars {
    name val
    ...
}

or for just one:

vars name val

[mholt]

@brucejo75 Does that vars directive and enhanced map directive satisfy your needs, then?

[brucejo75]

@mholt,

If it can do this, then yes! 👍 👍

  vars {
    SUB1 mysite.com
    SUB2 11
  }
  map {host} {port} {myvar} {
    "a.{SUB1}" {SUB2}11 "PORT{SUB2}11"
    "b.{SUB1}" {SUB2}22 "PORT{SUB2}11"
  }

Number substitution would be a bonus, just curious if that would work

Note: I think it may be a bit of an overkill. A simple pre-processor (similar to the way environment variables work) could probably accomplish all that I really want. So no placeholder lookup at runtime.

[mholt]

That should work now, yes. (In a route block)

[francislavoie]

Right now, vars is ordered after map, so the Caddyfile adapter would sort map to run before vars. So no, that example as-is would not work. 79cbe7b#diff-9bba53ca1b0eaab5c0f2bb3f964af37f393283d666c38d3436922f5ac66d92a6

[mholt]

I can reorder that, still pretty easy to do. 🤷‍♂️

Or maybe we could make it work like handle where it's the order it is in the Caddyfile...

And the vars might also need {http.vars.*}... hmm. Wonder if we should do more like what map does there.

[francislavoie]

Or maybe we could make it work like handle where it's the order it is in the Caddyfile...

That's not a thing, we still sort handle, but they're ordered according to their matchers (if any).

[brucejo75]

Question from my ignorant point of view:

Does the map directive reinterpret it's <input> and <output> arguments and recreate the table with each route evaluation?

I had assumed that there would be a step to create an internal representation of the map and then each runtime invocation of the map handler would use the "fixed" map parameters for its lookup. But all you are telling me would imply the map is reinterpreted on each route evaluation (which seems inefficient to me, probably not that big a deal in the scheme of things).

[francislavoie]

Does the map directive reinterpret it's <input> and <output> arguments and recreate the table with each route evaluation?

Yes, it's a request handler directive, so it runs for each request.

But all you are telling me would imply the map is reinterpreted on each route evaluation (which seems inefficient to me, probably not that big a deal in the scheme of things).

Yeah, it'll be basically instantaneous.

[mholt]

@brucejo75

Does the map directive reinterpret it's <input> and <output> arguments and recreate the table with each route evaluation?

To add to what Francis said, it's not recreating the table with each route evaluation, it's only performing a table lookup at each route evaluation, and ignores the rest of the table that's not being looked up. So it's really quite efficient.

Since I think your needs have been addressed in the latest changes, I will close the issue now, but feel free to continue discussion if necessary. (Again, you might have to use {http.vars.*} placeholders... and maybe we could make a {vars.*} shorthand for the Caddyfile. I do think it'd be kinda cool if it worked like map though.)