Too many headers http 400 Error while using revers proxies

[TheFuzz4]

The problem

Environment

• Home Assistant Core release with the issue: 115
• Last working Home Assistant Core release (if known): 114
• Operating environment (OS/Container/Supervised/Core): Container
• Integration causing this issue: pull request 38696
• Link to integration documentation on our website: PR 38696

Problem-relevant configuration.yaml

  trusted_proxies:
    - 192.168.7.1

Traceback/Error logs

2020-09-21 11:42:47 ERROR (MainThread) [homeassistant.components.http.forwarded] Too many headers for X-Forwarded-For: ['2607:fb90:6c82:27ae:994f:c5b0:3564:2d32', '108.162.216.49']
2020-09-21 11:42:47 ERROR (MainThread) [homeassistant.components.http.forwarded] Too many headers for X-Forwarded-For: ['2607:fb90:6c82:27ae:994f:c5b0:3564:2d32', '162.158.74.124']
2020-09-21 11:42:49 ERROR (MainThread) [homeassistant.components.http.forwarded] Too many headers for X-Forwarded-For: ['2607:fb90:6c82:27ae:994f:c5b0:3564:2d32', '162.158.75.187']

Additional information

In the past I've never had to provide the proxy addresses of Cloudflare as my Firewall only allows those IPs to come in. As Cloudflare does everything in subnet IPs do I need to put the full subnet into my configuration.yaml? Will it accept standard IP subneting for the reverse proxy addresses? i.e. 0.0.0.0/24

Thank you for your help with this item.

[TheFuzz4]

I believe this to be caused by this PR #38696

[probot-home-assistant]

http documentation
http source
_{^{(message by IssueLinks)}}

[tomashejatko]

Hello, I have the pretty same issue. My external access setup is a bit complicated tho: External client -> Cloudflare -> Haproxy -> Home Assistant instance.

My config is following (first is IP of haproxy):

  trusted_proxies:
    - 10.9.116.254
    - 173.245.48.0/20
    - 103.21.244.0/22
    - 103.22.200.0/22
    - 103.31.4.0/22
    - 141.101.64.0/18
    - 108.162.192.0/18
    - 190.93.240.0/20
    - 188.114.96.0/20
    - 197.234.240.0/22
    - 198.41.128.0/17
    - 162.158.0.0/15
    - 104.16.0.0/12
    - 172.64.0.0/13
    - 131.0.72.0/22

I have added all of CF ranges from https://www.cloudflare.com/ips/

But when I access HA from external site, I get these Too many headers for X-Forwarded-For errors in HA log

[nicklasl]

I got this issue after updating.
I run a local reverse proxy using nginx and get these errors:
Too many headers for X-Forwarded-For: ['192.168.1.1', '192.168.1.1'] where that IP is my router/dns.

Edit: my bad. My nginx config actually had a duplicate entry of the X-Forwarded-For header.

[TheFuzz4]

Yeah @tomashejatko your setup is identical to mine and you also are getting the same errors. This seems to be limited to those of us using CloudFlare. Hoping this gains some traction.

[frenck]

This is not a bug, but a configuration error in your reverse proxy configuration and is caused if the HTTP request that ends up with Home Assistant contains 2 or more X-Forwarded-For headers.

Basically, CloudFlare already sets an X-Forwarded-For header. Your proxy should ADD to the existing header provided by CloudFlare, not add a second header.

The problem is: If Home Assistant receives 2 or more X-Forwarded-For headers, which one should it use? <- that is why this was fixed and Home Assistant throws an error.

[tomashejatko]

From HAProxy documentation:

Since HAProxy works in reverse-proxy mode, the servers see its IP address as
their client address. This is sometimes annoying when the client's IP address
is expected in server logs. To solve this problem, the well-known HTTP header
"X-Forwarded-For" may be added by HAProxy to all requests sent to the server.
This header contains a value representing the client's IP address. Since this
header is always appended at the end of the existing header list, the server
must be configured to always use the last occurrence of this header only. See
the server's manual to find how to enable use of this standard header. Note
that only the last occurrence of the header must be used, since it is really
possible that the client has already brought one.

It really looks that haproxy send this header two times (I made little PHP script, it looked ok, but I am not sure if it will show duplicate headers or not)
So I made this workaround in haproxy config

option forwardfor header cf-forwarded-for

on my HASS backend, because I have option forwardfor in default and it cant be "turned off" via no option setting.

Hope this will help somebody

[frenck]

@tomashejatko For haproxy you should actually use option forwardfor append.

[tomashejatko]

I don't know if I miss something, but there is no "append" parameter, not even in latest version: https://cbonte.github.io/haproxy-dconv/2.3/configuration.html#4.2-option%20forwardfor - so the solution (more workaround IMHO) is to not use haproxy's X-Forwared-For header handling, as it will send it twice

[frenck]

Sorry, I should have looked it up, instead of speaking from top of my mind (which was incorrect).

http-request replace-value x-forwarded-for ^ "%[hdr(x-forwarded-for)], %[src]"

☝️ That is another solution that might put you into a workable direction.

[TheFuzz4]

So in my case I have 2 frontends in HA Proxy for HomeAssistant. One is for internal usage and the other for external. I turned off the x-forwarded-for on the external HA Proxy and now I'm cooking with gas. Thank you @frenck for the explanation.

[HelderFSFerreira]

Same error appear'd on my installation after upgrading to the latest version without any changes on my reverse proxy.
I'll update the proxy but shouldn't it be mentioned on breaking changes?

I know it's a miss configuration but some users can be affected.

[georgelza]

So in my case I have 2 frontends in HA Proxy for HomeAssistant. One is for internal usage and the other for external. I turned off the x-forwarded-for on the external HA Proxy and now I'm cooking with gas. Thank you @frenck for the explanation.

are you able to axxess home assistant from outside via the HA Mobile app on a phone ? I can get it working from outside in a browser (whichI'd actually want to disable, but want to get the mobile working.)

G

[flobernd]

@frenck While it's possible to work around that issue, I still think the Home Assistant webserver (and many other webserver implementations out there) fails to follow the de-facto standard for comma separated HTTP header values. For these headers both representations should be equivalent:

x-forwarded-for: A,B

<===>

x-forwarded-for: A
x-forwarded-for: B

From what I read, HA chooses to print this message to avoid ambivalence (as there seem to be problems in the past where HA used the wrong header). IMHO there shouldn't be "wrong" values as this header contains a chain. If the request goes through multipe proxies, the x-forwarded-for will either get appended to the existing header or a new x-forwarded-for is added. In either way, the first value should be the IP address of the actual client.

There is an informative comment in the HAProxy repo: haproxy/haproxy#44 (comment)

The "best" workaround is discussed in the HAProxy forums:
https://discourse.haproxy.org/t/appending-to-the-xff-header-does-not-work-as-expected/6415/2

This comment is particuarily interesting:

Be careful with this. By appending a value to an existing header instead of adding a new header, you’re giving the client all the keys to manipulate its contents and format, allowing it to be unparsable by the last server in the chain, effectively opening a security issue. While originally the extra header used to be a technical limitation, it’s now a reliability feature. If your server is having difficulties parsing header lists, it should be fixed instead of hacking on a header without prior checking that its contents will not cause trouble.

I really think this issue should be opened again.

[jonny190]

I also think this needs revisiting, Using Cloudflare then my PFsense router is causing me the same pain when trying to pass the real IP to home assistant rather then just the PFsense internal IP

[spcano01]

I don't really understand these things, but I finally figured it out! Again, had HAProxy reverse proxying everything just fine for some time. Then, several months ago - Home Assistant broke externally with the Too many headers error throwing up Cloudflare IPs and random LTE ones.

Finally figured it out, and here is the picture of my new action on my backend. I had to do it this way, because I use 2 frontends only, http & https with https having checked the "Use "forwardfor" option" box.

This was after trying multiple edits of copying and pasting what frenck wrote, and other threads all over. Hope this helps someone.

[mcarbonneaux]

Sorry, I should have looked it up, instead of speaking from top of my mind (which was incorrect).

http-request replace-value x-forwarded-for ^ "%[hdr(x-forwarded-for)], %[src]"

☝️ That is another solution that might put you into a workable direction.

in fact haproxy apply STRICTLY and on way (only multiple header) the rule of http rfc that multiple version of the same header is equivalent of the same header with multivalue separated by coma...

the x-forward is de facto standard... and most of the user of this standard reconnise only the version with coma...

is why i've exchangerd on this on the haproxy issue... because only haproxy force this way of doing...

the only solution is to not use option forwardfor but http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]".

[flobernd]

@mcarbonneaux Please read my comment above. HAProxy is NOT the problem here. It's the HASSIO webserver not following the HTTP standard. The devs just seem ignorant about this...

[mcarbonneaux]

@flobernd i'm not ok with that, haproxy must support the two posibility of the http rfc not only one (multiheader vs one header with multi value, the two are ok with the rfc), and be configurable.

and most of the application that support x-forward (quasi majority in fact) not support the multi header way of the http standard...

haproxy as reverse proxy must be abel to adapt to the destination not the reverse...

[flobernd]

@mcarbonneaux

and most of the application that support x-forward (quasi majority in fact) not support the multi header way of the http standard...

Even if that would be that case, issue is still on them. There is a HTTP standard for a reason and if you don't follow the standard (completely), you have to expect incompatibilities.

haproxy as reverse proxy must be abel to adapt to the destination not the reverse...

HAProxy does not know anything about the destination and as such it has to expect the destination server follows the HTTP standard.

HAProxy understands the comma separated header perfectly fine and chooses one of the valid actions to perform as a proxy: appending a new header. Reasons for that can e.g. be read up here: haproxy/haproxy#44 (comment). The HTTP standard nowhere mentions that a proxy must support both variants.

From a semantic perspective HAProxy does not alter the HTTP request at all!

Judging from this comment it's pretty clear that the HASSIO devs never checked the HTTP standard. There is no ambiguity as they state in this comment and there is a well defined way to handle such requests.

[bennydiamond]

Sorry, I should have looked it up, instead of speaking from top of my mind (which was incorrect).
http-request replace-value x-forwarded-for ^ "%[hdr(x-forwarded-for)], %[src]"
☝️ That is another solution that might put you into a workable direction.

in fact haproxy apply STRICTLY and on way (only multiple header) the rule of http rfc that multiple version of the same header is equivalent of the same header with multivalue separated by coma...

the x-forward is de facto standard... and most of the user of this standard reconnise only the version with coma...

is why i've exchangerd on this on the haproxy issue... because only haproxy force this way of doing...

the only solution is to not use option forwardfor but http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]".

Thank you. Your solution is unfortunately the right one. I guess it infers some more processing for haproxy to replace the header on every incoming requests.

Here's how the rule should look like in OPNSense HAproxy config page:

Add the rule to your "Public service" config for your frontend proxy.

[mcarbonneaux]

@flobernd

From a semantic perspective HAProxy does not alter the HTTP request at all!

when you use options forward haproxy add second x-forward-for headers to the http request.... is an alteration...

HAProxy does not know anything about the destination and as such it has to expect the destination server follows the HTTP standard.

i'm ok with that, and is why you must be abel to chose how haproxy apply the standard.... because not all backend are conform... specificaly about this aspect of the http standard (coma vs multiple header)...

The HTTP standard nowhere mentions that a proxy must support both variants.

yes but x-forward-for are not a real standard... but defacto standard...

and many interpretation has been done on the way of backend interpret it... and haproxy must be abel to address this different intepretation, even if is the default way of dooing of haproxy is ok in relation of the http standard... is not the only way of doing from the standard view...

and the http standard say coma or multiple header are equal... in that way he say that coma are valid too not juste multiple header...

but if the application that are in back of haproxy are generaly not completly strict with http standard... haproxy with this default way of doing make reverse proxification of this application generaly ko... haproxy as reverse proxy must be standard compliant (not only with multiple header way... must be compliant with the two way of the standard http...) but also must be working in majority of the way of backend work...

and in the x-forward defacto standard, in the majority of implementation are using coma... not multiple header... even it is partialy ok with the http standard...

the chance with haproxy it as many way to be configured, and the good solution to be compatible with majority of the usage of x-forward header defacto standard, is to not use option forwardfor but http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]".... that are ok with http standard and defacto x-forward standard general implementation...

@bennydiamond

Your solution is unfortunately the right one. I guess it infers some more processing for haproxy to replace the header on every incoming requests.

ok adding header in place of replacing modified one is less performant (but with autoscaling infrastructure he had very ligth impact, and with new generation of processor we speak in microseconds...), but is working with majority of backend...

is what i say to owner of haproxy... majority of the reverse proxy of the market (open source or proprietary) as chose this way of doing (replace and coma) because work by default... i arged that haproxy must have forward option with the two way of doing (the two are http compliant)...

[mcarbonneaux]

@flobernd

@mcarbonneaux Please read my comment above. HAProxy is NOT the problem here. It's the HASSIO webserver not following the HTTP standard. The devs just seem ignorant about this...

haproxy is partialy compliant with http standard because is conforming only on multiple header...
and hassio are also partialy compliant because is compliant with only coma way of the http standard....

but haproxy are reverse proxy and must abel to address the two way of the standard... because some backend chose one of them not the two... and i thing haproxy had chosed the performance way of doing, but not the more compatible way of doing...

[flobernd]

@mcarbonneaux

yes but x-forward-for are not a real standard... but defacto standard...

Doesn't matter. X-Forwarded-For is still a header and the semantics are defined for all headers (see latest official HTTP Semantics RFC9110).

haproxy is partialy compliant with http standard because is conforming only on multiple header...

I disagree. HAProxy "understands" (or ignores) both variants for the incomming request which means it's fully compliant. They just choose to append a new header vs. altering the original one, but this is perfectly fine as it does not change the semantics of the altered request. That's what I'm trying to explain to you 🙂

and hassio are also partialy compliant because is compliant with only coma way of the http standard....

For this part, I do agree.

---

I think there is no point in discussing this any further. I've already linked an official comment of one of the HAProxy members which describes the exact reasons for not using the comma separated list:

The reason for doing it the way it's done is because the alternative you propose is way slower : you first have to look for the last instance and only append to this instance, moving the tail. And possibly fold the other existing instances if any. For lots of people dealing with static servers or fast dynamic servers, wasting hundreds of nanoseconds looping on headers for ZERO added value (or to work around elementary server bugs) is not acceptable.

They instead suggest ...

... to rewrite the header using one http-request rule

I personally agree with them to use the extra header as their default behavior and at the end it's their decision! HAProxy has this exact behavior for 22 years now (wow!) and there is no need to change this. Especially as they provide a workaround with the header rewrite functionality.

HASSIO on the other hand has no good reason for not implementing both variants. Their explaination/excuse for not following the standard is just not valid. They wrote this:

The problem is: If Home Assistant receives 2 or more X-Forwarded-For headers, which one should it use? <- that is why this was fixed and Home Assistant throws an error.

They didn't know an answer to this question at that time, but it's well defined behavior as can be read up here.

If a HTTP server receives two headers with the same name, they simply get combined in the same order they are specified:

Example-Field: Foo, Bar
Example-Field: Baz

=>

Example-Field: Foo, Bar, Baz

In the special case of X-Forwarded-For this means the first IP in the chain should always be the IP of the original client and the last IP in the chain should always be the IP of the last proxy. Everything inbetween belongs to other proxy hops the request went through.

Maybe @frenck could give this another look? 🙂

[frenck]

Maybe @frenck could give this another look? 🙂

Thanks for pulling me into this conversation. It feels a bit demanding though...

To answer your question there, I have no interest in looking at it at this point.

This is old and closed as well.

../Frenck

[flobernd]

To answer your question there, I have no interest in looking at it at this point.

Sorry to hear that, but fair enough I guess...

This is old and closed as well.

This does not make it less relevant.

Would you accept PRs about this?

[mcarbonneaux]

I disagree. HAProxy "understands" (or ignores) both variants for the incomming request which means it's fully compliant. They just choose to append a new header vs. altering the original one, but this is perfectly fine as it does not change the semantics of the altered request. That's what I'm trying to explain to you 🙂

no I disagree, haproxy understand the inbound traffic in two way of the http standard, but when add x-forward he use only the way of adding header (not coma way) and is not simply configurable (only using http-request replace-header).

I disagree. HAProxy "understands" (or ignores) both variants for the incomming request which means it's fully compliant. They just choose to append a new header vs. altering the original one, but this is perfectly fine as it does not change the semantics of the altered request. That's what I'm trying to explain to you

no, i disagree, if haproxy support fully the http standard he must be configurable to use the two way of doing depend on the backend way of doing. is not to the backend to adapt to the reverse proxy but is the reverse proxy to adapt to the backend...

I personally agree with them to use the extra header as their default behavior and at the end it's their decision! HAProxy has this exact behavior for 22 years now (wow!) and there is no need to change this. Especially as they provide a workaround with the header rewrite functionality.

the problem of using another header... is that you must recode many application that work with this header... and is not the reverse proxy to force the way of doing, the reverse proxy must adapt to the backend... and must but configurable ...

In the special case of X-Forwarded-For this means the first IP in the chain should always be the IP of the original client and the last IP in the chain should always be the IP of the last proxy. Everything inbetween belongs to other proxy hops the request went through.

in real life they are no other header that are used in that way... only x-forward header... and is very special way of doing because are defacto standard....

if you whant to use the http standard you must use forwarded header in place of x-forward-*.... but none of the backend application know this header....

[unclesam87]

hey, i am quite frustrated and not sure if u could help me:
i tried to adjust my haproxy config to show me the real ips, but it wont work at all....(the comments are the section i allready tried and are still in there but maybe i made an misstake)

frontend SSL_Termination
    bind *:8443 ssl crt /etc/haproxy/certs/ strict-sni
    mode http
   # http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains;"
   # http-request add-header X-Forwarded-Proto https
   # option forwardfor
    http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]"
        acl https_home ssl_fc_sni fqdn.com
        use_backend home if `https_home``
backend home
    mode http
#    option forwardfor
#    http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]"
#    http-request add-header X-Forwarded-Proto https
#    http-request add-header X-Forwarded-Port 443
    server home_server 172.16.150.11:8123

and this is my home assistant config

http:
  use_x_forwarded_for: true
  trusted_proxies: 
    - 172.16.20.9
    - 172.16.20.0/24
    - 172.30.33.0/24

[Write]

hey, i am quite frustrated and not sure if u could help me: i tried to adjust my haproxy config to show me the real ips, but it wont work at all....(the comments are the section i allready tried and are still in there but maybe i made an misstake)

frontend SSL_Termination
    bind *:8443 ssl crt /etc/haproxy/certs/ strict-sni
    mode http
   # http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains;"
   # http-request add-header X-Forwarded-Proto https
   # option forwardfor
    http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]"
        acl https_home ssl_fc_sni fqdn.com
        use_backend home if `https_home``
backend home
    mode http
#    option forwardfor
#    http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]"
#    http-request add-header X-Forwarded-Proto https
#    http-request add-header X-Forwarded-Port 443
    server home_server 172.16.150.11:8123

and this is my home assistant config

http:
  use_x_forwarded_for: true
  trusted_proxies: 
    - 172.16.20.9
    - 172.16.20.0/24
    - 172.30.33.0/24

Not sure if it might help you but in my frontend I have this in this order :

    http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]"
    option forwardfor if-none

Above that I Have some more lines that indicate how to handle Cloudflare

    acl from_cf         src -f /usr/local/etc/haproxy/cloudflare/cloudflare_ips.lst
    acl cf_ip_hdr       req.hdr(CF-Connecting-IP) -m found
    http-request        set-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)] if from_cf cf_ip_hdr

where cloudflare_ips.lst is a file containing IPv4 and IPv6 range of IPs of Cloudflare servers

Which is basically just a text file containing an updated concatenation of https://www.cloudflare.com/ips-v4/# and https://www.cloudflare.com/ips-v6/#

In ha config, adding only the /32 IP of your HAProxy instance should be enough.