Add documentation for encode and file_server enhancements

[ueffel]

Adding documention for caddyserver/caddy#4045

Maybe the Response Matcher section should be more general, when caddyserver/caddy#4021 also gets merged?

[mholt]

Awesome, thanks so much for contributing these too. Really appreciate it, as will a lot of users!

Maybe the Response Matcher section should be more general, when caddyserver/caddy#4021 also gets merged?

Yeah, we can consider that for later possibly; I also don't just mind linking one to the other, or vice-versa.

[polarathene]

TL;DR: Any reason for encode.prefer and file_server.precompressed inconsistency? Why does encode require a prefer value to set a specific order when the equivalent file_server.precompressed does not?

Since the client can override preference by explicit weighted values and both client and server need to negotiate an encoding both support, why does encode need to repeat values to declare a desired order in prefer? Is there a situation where setting a specific preference would not be desirable?

---

Trying out the beta and looking at these docs (I don't think they're published officially yet?), could you please clarify if I understand this correctly?

Simple config using the two features (plus basic cache, having a placeholder for matching common mimetypes or file extensions to cache as sane default as well might be nice addition too 😉 ):

(cache-assets) {
  @assets {
    path_regexp \.(jpg|png|webp)$
  }

  header @assets Cache-Control "public, max-age=31536000"
}

(compress-assets) {
  # Uses sane defaults since v2.4, see docs.
  # Will encode dynamic content on-demand, so long as it wasn't compressed from a backend already.
  encode zstd gzip {
    # Needs to re-declare/repeat the order, `encode` order above has no relevance, 
    # `encode.prefer` however does, like `file_server.precompressed`.
    prefer zstd gzip
  } 
}

localhost {
  root * /srv/site
  file_server {
    # Serve static pre-compressed files if the compressed extension (zst/br/gz) exists beside the target file.
    # Declaration order here matters and behaves like `encode.prefer`, setting server preference provided client has none.
    precompressed zstd br gzip
  }

  import cache-assets
  import compress-assets
}

---

The encode directive needs prefer to repeat the actual order preferred server-side (provided the client doesn't provide any weighted preferences to override that), but precompressed implicitly is server-side preference?

This seems like an odd choice with possible redundancy? Once 2.4 is released officially, that would make deprecating prefer more of a hassle no? Is this inconsistency between encode and precompressed intentional?

Is there something I don't realize that the documentation has not made clear why encode requires explicit opt-in for server-side preference by repeating or slightly different order than what encode will already define?

---

Below is the related documentation in this PR. Apologies if I'm not seeing something obvious, but I figured it might be worth pointing out before 2.4 is released in case this inconsistency for related config was accidental:

prefer is the ordered list of enabled encoding formats to determine, which encoding to choose if the client has no strong preference (via q-factors in the Accept-Encoding header).
If prefer is not specified the first supported encoding from the Accept-Encoding header is used.

Enable Zstandard and Gzip compression and prefer Zstandard over Gzip:

encode zstd gzip {
	prefer zstd gzip
}

Without the prefer setting, a --compressed HTTP request via curl (meaning Accept-Encoding: deflate, gzip, br, zstd in curl >=7.72.0) would be served with Gzip encoding, because it is the first accepted encoding that both client and server support. With the prefer setting Zstandard encoding is served, because the client has no preference but the server (caddy) has.

• precompressed is the list of encoding formats to search for precompressed sidecar files.
• <formats...> is the ordered list of encoding formats to search for precompressed sidecar files. Supported formats are gzip, zstd and br.

If supported by the client (Accept-Encoding header) checks the existence of precompressed files along side the requested file. So if /path/to/file is requested, it checks for /path/to/file.zst, /path/to/file.br and /path/to/file.gz in that order and serves the first available file with corresponding Content-Encoding:

file_server {
	precompressed zstd br gzip
}

[polarathene]

I looked at the present Caddy 2.3 encode directive docs and read over the original issue by @ueffel for this feature.

From what I can make out, encode and encode.prefer expect the same input as a list of encoders, at least within the Caddyfile. JSON treats the two differently with encodings object and prefer array/list. I assume that's the reason? Preference order could be set on the encodings object if that isn't suitable to convert to a list, or if the Caddyfile could implicitly set encode format list to be prefer that would make sense too?

There are no config examples for adjusting the encoders, just a mentioned setting for gzip compression level which seems to differ in approach between Caddyfile and JSON somewhat?

Is it too much trouble to have prefer inherit the encode list for a Caddyfile by default?

[ueffel]

Problem is, that the JSON-Object is marshalled to a Go-Map and these have to no guarantee on the order of keys when iterating over the map. Keys generally won't be in insertion order.

The prefer was introduced to handle configurations like this:

encode {
    zstd
    gzip 5
}

It's not really clear which encoding is preferred.

Another point is backwards compatibility. An implicit prefer order from encode zstd gzip may behave differently to when no prefer order is set.

Of cause It would be possible to implicitly set the prefer order in cases of easy configs (encode zstd gzip). But it is not for me to decide if that should be implemented.

[polarathene]

EDIT: Raised an issue summarizing this comment better.

---

TL;DR: I don't see any backwards compatibility concern since default content to encode has also changed. If you can implicitly set prefer based on the list declared in existing encode that seems preferable to support.

Hopefully @mholt agrees. It's a minor UX improvement, and I know that JSON config is encouraged over the Caddyfile, but still seems worthwhile for consistency with the new precompressed setting and keeping config simple instead of unnecessary verbosity.

---

Lengthy response (you can ignore it, I'll provide a more terse/focused response again if relevant)

Problem is, that the JSON-Object is marshalled to a Go-Map and these have to no guarantee on the order of keys when iterating over the map. Keys generally won't be in insertion order.

I'm not a Go dev, so I assume that means that despite the docs stating the format for encode and prefer are the same list of encoders, that there's no way to read that order when converting to JSON config? Otherwise you can append an order as keys or implicit prefer order.

Personally it seems odd to require what for most may seem redundant, along with the inconsistency with precompressed from a user point of view in the Caddyfile config. If it's not something you have any control over, I understand that.

---

The prefer was introduced to handle configurations like this:

While I don't know the internals of Caddyfile to JSON logic, I always just saw that as implicitly enabling (as is the case with zstd in the example) an encoding while also making it possible to configure the object (zstd doesn't support any config presently afaik, unlike gzip?).

Is it invalid to also specify the encoders like:

encode zstd gzip {
  gzip 5
}

As that supposedly would declare a preference order just as well as prefer while allowing you to pass any non-default config to an encoder that Caddy supports. If so I'm not sure what value prefer provides as a setting other than being more explicit and having a more direct mapping to the JSON representation?

It's not really clear which encoding is preferred.

Keeping prefer is fine for that, as preference has been vague prior to 2.4 without it.

My question was if it is possible to defer preference from the encode zstd gzip {} declaration before it is converted to the JSON config object for each encoder. Can you access the list order and modify the JSON to implicitly add prefer? (that was answered at the end of the response)

---

Another point is backwards compatibility. An implicit prefer order from encode zstd gzip may behave differently to when no prefer order is set.

Was there ever a guarantee on this? From your issue on the topic it seems like this was unexpected as a user to discover how selection was working (you could not negotiate for zstd from cURL if gzip was declared first). Prior behaviour wasn't documented was it?

As only zstd and gzip are supported officially, and no web browser officially supports zstd, how many users would be affected? Present behaviour was basically always negotiating gzip if the client made it available? If the admin explicitly restricted it to zstd or gzip, then no compatibility issue exists?

The prefer feature itself along with sane defaults for file types is v2.4 only, and AFAIK docs aren't presently versioned? If you're on an older version, you could consult the docs to find mismatch in config behaviour already due to fixes/improvements that came with future releases.

There are release notes for anyone upgrading that needs to be aware of any potential concerns (this feature is already enabling a restricted set of default file types instead of the previous default of anything goes where I noticed compressed assets like webp were going through gzip), I don't see "backwards compatibility" as being relevant here due to these related changes.

---

Of cause It would be possible to implicitly set the prefer order in cases of easy configs (encode zstd gzip). But it is not for me to decide if that should be implemented.

I understand that. That would be a decision for @mholt ? One of the parts about Caddy I enjoy over nginx is that for those who want to use Caddyfile config is much simpler with smart defaults.

I think the prefer setting is useful if a user needs to be explicit, but has ample opportunity for being implicit instead of more verbose/repetitive. It would be best to address this prior to the 2.4 release though if you agree?

[francislavoie]

@polarathene I appreciate that you're interested, and your feedback is valuable. But I do have to say that these walls of text are a huge burden for us. I takes a lot of effort to read through and potentially respond to each of your points. From a cost-benefit perspective, it feels really difficult to engage here. I'm saying this now because it's not the first time that I think @mholt or I have felt overwhelmed by these types of comments (e.g. the auto-https discussion) so I'd just like to ask to try to be more concise in the future.

Anyways, I don't have a strong opinion either way, but I do agree we should make a decision on this before v2.4.0 stable, but we also can't afford to make this block that release (probably next week Monday ish).

[polarathene]

@francislavoie I understand, I apologize about the verbosity and will try to keep it terse in future. The TLDR at the top was an effort to address that.

There's no need to block the release.

• prefer remains useful in the JSON config.
• It's a minor improvement for Caddyfile. No rush.