413 Request Entity Too Large #21

rawlingsj · 2016-04-25T12:41:44Z

We've deployed a docker registry and created an ingress rule to it's kubernetes service whilst using the nginx ingress controller. When pushing larger images we quickly hit the nginx limits giving us the error below.

Error parsing HTTP response: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.9.14</center>\r\n</body>\r\n</html>\r\n"

I've forked the repo and hacked the nginx config adding a client_max_body_size attribute so we can push larger images. For a proper solution though, it might be nice to set a value in the kubernetes ingress rule and have that used when the nginx controller is updated?

The text was updated successfully, but these errors were encountered:

pleshakov · 2016-04-25T17:34:43Z

We plan to add the ability to configure the global NGINX parameters through a ConfigMap.

To allow to customize the configuration per Ingress resource, we can leverage annotations. As an example,

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    nginx/client_max_body_size: 1m
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

Will that work for your case?

rawlingsj · 2016-04-25T17:44:59Z

@pleshakov yeah that would be great!

styxlab · 2016-05-31T22:12:17Z

This already works with ConfigMaps such as:

apiVersion: v1
kind: ConfigMap
data:
  body-size: "0"
metadata:
  name: nginx-conf

A value of "0" lifts the restriction on client_max_body_size.
Make sure to reference the ConfigMap in your ingress controller like so:

args:
        - /nginx-ingress-controller
        - --default-backend-service=default/default-http-backend
        - --nginx-configmap=default/nginx-conf

For more information on the available config parameters have a look at here.

pleshakov · 2016-06-01T22:48:56Z

We're adding this features as well as other NGINX configuration parameters soon. Stay tuned.

pleshakov · 2016-06-16T00:23:14Z

customization of NGINX configuration was added in #33

pleshakov · 2016-06-16T00:25:39Z

@rawlingsj Please check the example on how to customize NGINX: https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/customization

Hronom · 2017-02-22T23:12:13Z

Not worked in gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2 and gcr.io/google_containers/nginx-ingress-controller:0.8.3 I set value like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/client-max-body-size: "100m"
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: test
          servicePort: 8888

pleshakov · 2017-02-23T12:18:21Z

@Hronom the google_containers/nginx-ingress-controller images belong to a different project. Please see the documentation for that project available on this page https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md

The image for this project -- https://hub.docker.com/r/nginxdemos/nginx-ingress/tags/

Hronom · 2017-02-23T12:23:19Z

@pleshakov sorry guys, too many kubernetes ingresses over internet...

misteruly · 2017-08-28T08:56:29Z

@Hronom I'm setting it like that, but it doesn't work.don't know why??

benbonnet · 2017-11-27T10:56:44Z

In case it helps, 413 solved with the ingress.kubernetes.io/proxy-body-size :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: services
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/proxy-body-size: 50m

Running quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17

lastcoolnameleft · 2018-07-28T16:18:04Z

FYI, the annotation has changed and is now:

    nginx.ingress.kubernetes.io/proxy-body-size: 50m

Also, I had to restart the nginx pod for the effect to take place. It immediately started working after that.

ryanrca · 2019-02-02T07:48:27Z

Setting it to "0" makes the nginx post size unrestricted:

    nginx.ingress.kubernetes.io/proxy-body-size: "0"

mehmetali · 2019-07-23T07:13:34Z

Looks like it changed to client-max-body-size for configmap.
client-max-body-size: "0"

I didn't try with annotations:
nginx.org/client-max-body-size: "0"

AntonioArts · 2019-08-14T11:49:28Z

For apiVersion: extensions/v1beta1
nginx.ingress.kubernetes.io/proxy-body-size: 50m annotation didn't solve 413 error for me.
When I've added one more annotation
nginx.org/client-max-body-size: "50m" it did the job

alexellis · 2019-08-14T17:22:04Z

I came here to configure the Docker registry helm chart and found:

    nginx.ingress.kubernetes.io/proxy-body-size: 50m

I added it to the Ingress for the registry and it did the trick. 👌

theguyver · 2019-11-17T18:52:04Z

This solved it for me:
nginx.ingress.kubernetes.io/proxy-body-size: 50m

Ganeshkumar1023 · 2019-11-19T01:57:36Z

FYI, the annotation has changed and is now:
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
Also, I had to restart the Nginx pod for the effect to take place. It immediately started working after that.

Hi,

I know this is a very old post, we have the same issue and I've updated nginx.ingress.kubernetes.io/proxy-body-size: 50m in the file, now I have to restart Nginx POD to changes to effect, can you please let me know how to restart K8 Nginix POD ..?

I know this way, we can restart PODs,

kubectl scale deployment name--replicas=0 -n service
kubectl scale deployment name --replicas=1 -n service

any other way i can restart Nginx PODs to reload the configuration.

techsaau · 2020-02-13T11:23:09Z

@Ganeshkumar1023 in latest version of kubectl (1.17.0) they have added restart sub-command under rollout command. You can use this to restart the pods. so command would be:

kubectl rollout restart deployment/abc
kubectl rollout restart -f abc.yaml

Aech1977 · 2020-03-11T11:46:30Z

I have tried both proxy-body-size and client-max-body-size on the configmap and did a rolling restart of the nginx controller pods and when I grep the nginx.conf file in the pod it returns the default 1m. I am trying to do this within Azure Kubernetes Service (AKS). I'm working with someone from their support. They said its not on them since it appears to be a nginx config issue.

The weird hting is we had other clusters in Azure that this wasnt an issue on until we discovered this with some of the newer deployments. The initial fix they came up with is what is in this thread but it just refuses to change.

Below is my configmap:

apiVersion: v1
data:
  client-max-body-size: 0m
  proxy-connect-timeout: 10s
  proxy-read-timeout: 10s
kind: ConfigMap
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"nginx-nginx-ingress-controller-7b9bff87b8-vxv8q","leaseDurationSeconds":30,"acquireTime":"2020-03-10T20:52:06Z","renewTime":"2020-03-10T20:53:21Z","leaderTransitions":1}'
  creationTimestamp: "2020-03-10T18:34:01Z"
  name: ingress-controller-leader-nginx
  namespace: ingress-nginx
  resourceVersion: "23928"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx
  uid: b68a2143-62fd-11ea-ab45-d67902848a80

After issuing a rolling restart: kubectl rollout restart deployment/nginx-nginx-ingress-controller -n ingress-nginx

Grepping the nginx ingress controller pod to query the value now reveals:

kubectl exec -n ingress-nginx nginx-nginx-ingress-controller-7b9bff87b8-p4ppw cat nginx.conf | grep client_max_body_size
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    21m;

Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

alexellis · 2020-03-11T12:23:27Z

@Waterdrips cc

misteruly · 2020-03-12T02:24:12Z

think you

…

------------------ 原始邮件 ------------------ 发件人: "Aech1977"<[email protected]>; 发送时间: 2020年3月11日(星期三) 晚上7:46 收件人: "nginxinc/kubernetes-ingress"<[email protected]>; 抄送: "蓝鹏"<[email protected]>;"Comment"<[email protected]>; 主题: Re: [nginxinc/kubernetes-ingress] 413 Request Entity Too Large (#21) I have tried both proxy-body-size and client-max-body-size on the configmap and did a rolling restart of the nginx controller pods and when I grep the nginx.conf file in the pod it returns the default 1m. I am trying to do this within Azure Kubernetes Service (AKS). I'm working with someone from their support. They said its not on them since it appears to be a nginx config issue. The weird hting is we had other clusters in Azure that this wasnt an issue on until we discovered this with some of the newer deployments. The initial fix they came up with is what is in this thread but it just refuses to change. Below is my configmap: Please edit the object below. Lines beginning with a '#' will be ignored, and an empty file will abort the edit. If an error occurs while saving this file will be reopened with the relevant failures. apiVersion: v1 data: client-max-body-size: 0m proxy-connect-timeout: 10s proxy-read-timeout: 10s kind: ConfigMap metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"nginx-nginx-ingress-controller-7b9bff87b8-vxv8q","leaseDurationSeconds":30,"acquireTime":"2020-03-10T20:52:06Z","renewTime":"2020-03-10T20:53:21Z","leaderTransitions":1}' creationTimestamp: "2020-03-10T18:34:01Z" name: ingress-controller-leader-nginx namespace: ingress-nginx resourceVersion: "23928" selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx uid: b68a2143-62fd-11ea-ab45-d67902848a80 After issuing a rolling restart: kubectl rollout restart deployment/nginx-nginx-ingress-controller -n ingress-nginx Grepping the nginx ingress controller pod to query the value now reveals: kubectl exec -n ingress-nginx nginx-nginx-ingress-controller-7b9bff87b8-p4ppw cat nginx.conf | grep client_max_body_size client_max_body_size 1m; client_max_body_size 1m; client_max_body_size 1m; client_max_body_size 1m; client_max_body_size 21m; Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

jomel · 2020-10-19T21:02:37Z

Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

@Aech1977 did you solve it?

jvirriel-per · 2020-10-28T17:22:17Z

@Aech1977 this key is wrong client-max-body-size: 0m
the key for the configmap is proxy-body-size: 0m

# This is the 1st commit message: add ingress mtls test # This is the commit message #2: add std vs # This is the commit message #3: change vs host # This is the commit message #4: Update tls secret # This is the commit message #5: update certs with host # This is the commit message #6: modify get_cert # This is the commit message #7: Addind encoded cert # This is the commit message #8: Update secrets # This is the commit message #9: Add correct cert and SNI module # This is the commit message #10: Bump styfle/cancel-workflow-action from 0.8.0 to 0.9.0 (#1527) Bumps [styfle/cancel-workflow-action](https://github.com/styfle/cancel-workflow-action) from 0.8.0 to 0.9.0. - [Release notes](https://github.com/styfle/cancel-workflow-action/releases) - [Commits](styfle/cancel-workflow-action@0.8.0...89f242e) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> # This is the commit message #11: Remove patch version from Docker image for tests (#1534) # This is the commit message #12: Add tests for Ingress TLS termination # This is the commit message #13: Improve assertion of TLS errors in tests When NGINX terminates a TLS connection for a server with a missing/invalid TLS secret, we expect NGINX to reject the connection with the error TLSV1_UNRECOGNIZED_NAME In this commit we: * ensure the specific error * rename the assertion function to be more specific # This is the commit message #14: Bump k8s.io/client-go from 0.20.5 to 0.21.0 (#1530) Bumps [k8s.io/client-go](https://github.com/kubernetes/client-go) from 0.20.5 to 0.21.0. - [Release notes](https://github.com/kubernetes/client-go/releases) - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.20.5...v0.21.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> # This is the commit message #15: Improve tests Dockerfile * Reorganize layers so that changes to the tests do not cause a full image rebuilt * Use .dockerignore to ignore cache folders * Convert spaces to tabs for consistency with the other Dockerfiles # This is the commit message #16: Upgrade kubernetes-python client to 12.0.1 (#1522) * Upgrade kubernetes-python client to 12.0.1 Co-authored-by: Venktesh Patel <[email protected]> # This is the commit message #17: Bump k8s.io/code-generator from 0.20.5 to 0.21.0 (#1531) Bumps [k8s.io/code-generator](https://github.com/kubernetes/code-generator) from 0.20.5 to 0.21.0. - [Release notes](https://github.com/kubernetes/code-generator/releases) - [Commits](kubernetes/code-generator@v0.20.5...v0.21.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> # This is the commit message #18: Test all images (#1533) * Test on all images * Update nightly to test all images * Run all test markers on debian plus also * Update .github/workflows/nightly.yml # This is the commit message #19: Add tests for default server # This is the commit message #20: Support running tests in kind # This is the commit message #21: Update badge for Fossa (#1546) # This is the commit message #22: Fix ensuring connection in tests * Add timeout for establishing a connection to prevent potential "hangs" of the test runs. The problem was noticeable when running tests in kind. * Increase the number of tries to make sure the Ingress Controller pod has enough time to get ready. When running tests in kind locally the number of tries sometimes was not enough. # This is the commit message #23: Ensure connection in Ingress TLS tests Ensure connection to NGINX before running tests. Without ensuring, sometimes the first connection to NGINX would hang (timeout). The problem is noticable when running tests in kind. # This is the commit message #24: Revert changes in nightly for now (#1547) # This is the commit message #25: Bump actions/cache from v2.1.4 to v2.1.5 (#1541) Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...1a9e213) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> # This is the commit message #26: Create release workflow

pleshakov mentioned this issue May 12, 2016

Customization of NGINX configuration #27

Closed

rawlingsj mentioned this issue Jun 1, 2016

implement a more A/B testing style rolling upgrade using Ingress or Traefik? fabric8io/fabric8#6049

Open

pleshakov mentioned this issue Jun 15, 2016

Added customization of NGINX Configuration #33

Merged

pleshakov closed this as completed Jun 16, 2016

minhptruong2010 mentioned this issue Jul 10, 2018

Ingress paths other than root does not work. #304

Closed

rhaas80 mentioned this issue Mar 14, 2019

request entity too large error when trying to save notebook nds-org/jupyter-et#8

Closed

Mofef mentioned this issue Jun 18, 2019

docs/integrations/nginx.md annotations seem to be missing "nginx." as a prefix to the keys polyaxon/polyaxon#476

Closed

LeandroPT mentioned this issue Aug 1, 2019

docker push keeps retrying and failing to local registry distribution/distribution#2042

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

413 Request Entity Too Large #21

413 Request Entity Too Large #21

rawlingsj commented Apr 25, 2016

pleshakov commented Apr 25, 2016

rawlingsj commented Apr 25, 2016

styxlab commented May 31, 2016 •

edited

Loading

pleshakov commented Jun 1, 2016

pleshakov commented Jun 16, 2016

pleshakov commented Jun 16, 2016

Hronom commented Feb 22, 2017

pleshakov commented Feb 23, 2017

Hronom commented Feb 23, 2017

misteruly commented Aug 28, 2017 •

edited

Loading

benbonnet commented Nov 27, 2017 •

edited

Loading

lastcoolnameleft commented Jul 28, 2018

ryanrca commented Feb 2, 2019 •

edited

Loading

mehmetali commented Jul 23, 2019

AntonioArts commented Aug 14, 2019

alexellis commented Aug 14, 2019 •

edited

Loading

theguyver commented Nov 17, 2019

Ganeshkumar1023 commented Nov 19, 2019 •

edited

Loading

techsaau commented Feb 13, 2020 •

edited

Loading

Aech1977 commented Mar 11, 2020 •

edited

Loading

alexellis commented Mar 11, 2020

misteruly commented Mar 12, 2020 via email

jomel commented Oct 19, 2020

jvirriel-per commented Oct 28, 2020

413 Request Entity Too Large #21

413 Request Entity Too Large #21

Comments

rawlingsj commented Apr 25, 2016

pleshakov commented Apr 25, 2016

rawlingsj commented Apr 25, 2016

styxlab commented May 31, 2016 • edited Loading

pleshakov commented Jun 1, 2016

pleshakov commented Jun 16, 2016

pleshakov commented Jun 16, 2016

Hronom commented Feb 22, 2017

pleshakov commented Feb 23, 2017

Hronom commented Feb 23, 2017

misteruly commented Aug 28, 2017 • edited Loading

benbonnet commented Nov 27, 2017 • edited Loading

lastcoolnameleft commented Jul 28, 2018

ryanrca commented Feb 2, 2019 • edited Loading

mehmetali commented Jul 23, 2019

AntonioArts commented Aug 14, 2019

alexellis commented Aug 14, 2019 • edited Loading

theguyver commented Nov 17, 2019

Ganeshkumar1023 commented Nov 19, 2019 • edited Loading

techsaau commented Feb 13, 2020 • edited Loading

Aech1977 commented Mar 11, 2020 • edited Loading

alexellis commented Mar 11, 2020

misteruly commented Mar 12, 2020 via email

jomel commented Oct 19, 2020

jvirriel-per commented Oct 28, 2020

styxlab commented May 31, 2016 •

edited

Loading

misteruly commented Aug 28, 2017 •

edited

Loading

benbonnet commented Nov 27, 2017 •

edited

Loading

ryanrca commented Feb 2, 2019 •

edited

Loading

alexellis commented Aug 14, 2019 •

edited

Loading

Ganeshkumar1023 commented Nov 19, 2019 •

edited

Loading

techsaau commented Feb 13, 2020 •

edited

Loading

Aech1977 commented Mar 11, 2020 •

edited

Loading