-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lua VM crashed, reason: not enough memory #750
Comments
Just a few questions to better understand the context:
|
It's the Kong .deb package, running on Ubuntu Vivid. Nginx config is unchanged - the only thing I touched in kong.yml is the Cassandra hostnames. I've added the key-auth and mashape-analytics plugins via the API. I'm also using rate limiting. There are a few thousand consumers registered, each with a rate limit and one or two keys. Let me know if there's any more helpful info I can provide. |
It is likely to me that the analytics plugin cannot buffer the amount of data at this pace and thus is causing the crash. This is similar to openresty/lua-nginx-module#148 and openresty/lua-nginx-module#230 which propose ways of investigating the issue. For now:
I am not sure it is the reason but it is what seems more likely to me. |
I am not sure this can be qualified as a bug since memory limitations are, granted, optimizable, but ultimately quite unavoidable. This needs more investigation (especially finding out if it can be optimized), before being qualified as so. The last question ( |
If we confirm the problem is the Galileo in-memory batch growing too big, this could be qualified as an enhancement for the plugin and a solution could be dumping the buffer on disk when it gets bigger than a specific size (kinda to how nginx buffers request/response bodies to file if they are too big). |
@thibaultcha My theory at the moment is, yes, it's the analytics plugin. I turned it off for a few hours today and didn't see a single error of this kind. I turned it on again just before I went home, so I'll check the logs again tomorrow and see if the errors came back.
No, body logging is disabled, and the requests are all GETs anyway
100 events and 2 seconds (i.e. the default values)
Just 1 API
I don't have Kong's debug logs enabled, so it's hard to see exactly when it is sending batches. The Galileo dashboard does show data, but it has huge gaps of an hour or so at a time. If I'm reading the source of the analytics plugin right, the size of a single batch is limited but the queue of batches waiting to be sent is unbounded. This would explain the out-of-memory issue, if for any reason Kong was temporarily unable to connect to the Analytics server. @thefosk 's suggestion of spooling to disk sounds good, but it also sounds quite complex and thus unlikely to be implemented for a while. As a first step I would be quite happy with:
I'm much happier losing analytics data for 100 events in a controlled manner than having Kong run out of memory, blow up without warning, and lose the data for the entire waiting-to-send queue. |
Even if you don't have debug logs enabled, the error.log file will have errors logs when batches were not acknowledged by Galileo. It would be very useful here to know if any of those happen, and what is the status code returned by Galileo. Those informations are available in the logs. Currently, batches enqueued are dropped if Galileo is responding with a 400 status code, but it might have been changed on Galileo's side and other status codes could now be used, causing an invalid batch to not be dropped. This could be one reason. Having a limit in the queue of batches is also a possibility, but it might not be needed if the error is coming from un-dropped invalid batches. |
I've just checked error.log again and found some errors coming from the plugin. There are quite a few of these per day, but much fewer than the number of
These also appeared, but only 8 of them over one 5 minute period:
Is it possible that the Analytics server is not returning (many) error responses but is instead taking a long time to respond to the requests? That would be consistent with the behaviour we are seeing, i.e. out-of-memory errors + not many error logs. The 504 responses being returned also suggest that the Analytics server might be struggling to handle the load. |
Sorry to repeat myself, but I strongly recommend having a bound on that queue. Relying on the HTTP response code is not enough - if the analytics server responds slowly (as I suspect it is doing), or if it goes down for a few minutes, as it inevitably will now and again, then the queue will start to grow. It's a simple matter of fault isolation: auxiliary functionality such as the analytics plugin should not be able to cause any adverse effect on the main functionality of Kong, no matter what happens. And an out-of-memory error crashing the whole process is the mother of all adverse effects :) I would really appreciate a quick fix for this. I was planning to put Kong into production this week, but I won't be able to do that, at least not with the analytics plugin enabled, until this issue is fixed. And since we are paying for Galileo, disabling the analytics would be a real shame. |
I said "it might" not be needed, in the sense that the issue could have not existed if Kong was handling I have done so, as well as other improvements for the buffer in #757, which is opened against master as a hotfix. I am hoping to finish testing it soon, so we can release a new version ( |
As reported by #750, the buffer is vulnerable to a heavy load because of its `sending_queue` of batches pending for sending. It also does not handle the new 207 HTTP status code returned by the collector in case of invalid ALFs. Changes: - handle 207 HTTP status code by discarding the batch. Some ALFs in it will have been saved, and the invalid one(s) should not be retried. - implement a maximum size (in MB) for the sending_queue, as suggested by #750. Originally, I was about to implement such a size limitation by "number of batches pending", but it would not be intuitive for users to know what fits best their use case, because ALF sizes varies from one API to another, and one endpoint to another. By defining it in MB it is easier for users to chose a value. The default value of 10MB has been chosen after performing some benchmarking, and should handle from 300 to 500 req/s depending on the ALFs sizes. When the `sending_queue` has reached its limit, the current ALFs in the buffer will be **discarded**. - implement a retry policy. Instead of insisting on retrying to send batches when the collector cannot be reached, a delay is computed an exponentially increases on each failure to connect to the collector. That delay is shared by all workers. This avoids to load the collector when it is having difficulties and saves up bandwidth on Kong's side. As soon as the collector can be reached again, the delay is reset. Currently, the minimum retry delay is 1s and the maximum is 60s. Those values cannot be configured. - no more line jumps in logs printing responses from the collector.
Awesome, thanks for the quick response! |
### Summary #### 2.6.0 ``` Release 2.6.0 Tue February 6 2024 Security fixes: #789 #814 CVE-2023-52425 -- Fix quadratic runtime issues with big tokens that can cause denial of service, in partial where dealing with compressed XML input. Applications that parsed a document in one go -- a single call to functions XML_Parse or XML_ParseBuffer -- were not affected. The smaller the chunks/buffers you use for parsing previously, the bigger the problem prior to the fix. Backporters should be careful to no omit parts of pull request #789 and to include earlier pull request #771, in order to not break the fix. #777 CVE-2023-52426 -- Fix billion laughs attacks for users compiling *without* XML_DTD defined (which is not common). Users with XML_DTD defined have been protected since Expat >=2.4.0 (and that was CVE-2013-0340 back then). Bug fixes: #753 Fix parse-size-dependent "invalid token" error for external entities that start with a byte order mark #780 Fix NULL pointer dereference in setContext via XML_ExternalEntityParserCreate for compilation with XML_DTD undefined #812 #813 Protect against closing entities out of order Other changes: #723 Improve support for arc4random/arc4random_buf #771 #788 Improve buffer growth in XML_GetBuffer and XML_Parse #761 #770 xmlwf: Support --help and --version #759 #770 xmlwf: Support custom buffer size for XML_GetBuffer and read #744 xmlwf: Improve language and URL clickability in help output #673 examples: Add new example "element_declarations.c" #764 Be stricter about macro XML_CONTEXT_BYTES at build time #765 Make inclusion to expat_config.h consistent #726 #727 Autotools: configure.ac: Support --disable-maintainer-mode #678 #705 .. #706 #733 #792 Autotools: Sync CMake templates with CMake 3.26 #795 Autotools: Make installation of shipped man page doc/xmlwf.1 independent of docbook2man availability #815 Autotools|CMake: Add missing -DXML_STATIC to pkg-config file section "Cflags.private" in order to fix compilation against static libexpat using pkg-config on Windows #724 #751 Autotools|CMake: Require a C99 compiler (a de-facto requirement already since Expat 2.2.2 of 2017) #793 Autotools|CMake: Fix PACKAGE_BUGREPORT variable #750 #786 Autotools|CMake: Make test suite require a C++11 compiler #749 CMake: Require CMake >=3.5.0 #672 CMake: Lowercase off_t and size_t to help a bug in Meson #746 CMake: Sort xmlwf sources alphabetically #785 CMake|Windows: Fix generation of DLL file version info #790 CMake: Build tests/benchmark/benchmark.c as well for a build with -DEXPAT_BUILD_TESTS=ON #745 #757 docs: Document the importance of isFinal + adjust tests accordingly #736 docs: Improve use of "NULL" and "null" #713 docs: Be specific about version of XML (XML 1.0r4) and version of C (C99); (XML 1.0r5 will need a sponsor.) #762 docs: reference.html: Promote function XML_ParseBuffer more #779 docs: reference.html: Add HTML anchors to XML_* macros #760 docs: reference.html: Upgrade to OK.css 1.2.0 #763 #739 docs: Fix typos #696 docs|CI: Use HTTPS URLs instead of HTTP at various places #669 #670 .. #692 #703 .. #733 #772 Address compiler warnings #798 #800 Address clang-tidy warnings #775 #776 Version info bumped from 9:10:8 (libexpat*.so.1.8.10) to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/ for what these numbers do Infrastructure: #700 #701 docs: Document security policy in file SECURITY.md #766 docs: Improve parse buffer variables in-code documentation #674 #738 .. #740 #747 .. #748 #781 #782 Refactor coverage and conformance tests #714 #716 Refactor debug level variables to unsigned long #671 Improve handling of empty environment variable value in function getDebugLevel (without visible user effect) #755 #774 .. #758 #783 .. #784 #787 tests: Improve test coverage with regard to parse chunk size #660 #797 #801 Fuzzing: Improve fuzzing coverage #367 #799 Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests #698 #721 CI: Resolve some Travis CI leftovers #669 CI: Be robust towards absence of Git tags #693 #694 CI: Set permissions to "contents: read" for security #709 CI: Pin all GitHub Actions to specific commits for security #739 CI: Reject spelling errors using codespell #798 CI: Enforce clang-tidy clean code #773 #808 .. #809 #810 CI: Upgrade Clang from 15 to 18 #796 CI: Start using Clang's Control Flow Integrity sanitizer #675 #720 #722 CI: Adapt to breaking changes in GitHub Actions Ubuntu images #689 CI: Adapt to breaking changes in Clang/LLVM Debian packaging #763 CI: Adapt to breaking changes in codespell #803 CI: Adapt to breaking changes in Cppcheck Special thanks to: Ivan Galkin Joyce Brum Philippe Antoine Rhodri James Snild Dolkow spookyahell Steven Garske and Clang AddressSanitizer Clang UndefinedBehaviorSanitizer codespell GCC Farm Project OSS-Fuzz Sony Mobile ``` #### 2.6.1 ``` Release 2.6.1 Thu February 29 2024 Bug fixes: #817 Make tests independent of CPU speed, and thus more robust #828 #836 Expose billion laughs API with XML_DTD defined and XML_GE undefined, regression from 2.6.0 Other changes: #829 Hide test-only code behind new internal macro #833 Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P #819 Address compiler warnings #832 #834 Version info bumped from 10:0:9 (libexpat*.so.1.9.0) to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/ for what these numbers do Infrastructure: #818 CI: Adapt to breaking changes in clang-format Special thanks to: David Hall Snild Dolkow ``` #### 2.6.2 ``` Release 2.6.2 Wed March 13 2024 Security fixes: #839 #842 CVE-2024-28757 -- Prevent billion laughs attacks with isolated use of external parsers. Please see the commit message of commit 1d50b80cf31de87750103656f6eb693746854aa8 for details. Bug fixes: #839 #841 Reject direct parameter entity recursion and avoid the related undefined behavior Other changes: #847 Autotools: Fix build for DOCBOOK_TO_MAN containing spaces #837 Add missing #821 and #824 to 2.6.1 change log #838 #843 Version info bumped from 10:1:9 (libexpat*.so.1.9.1) to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/ for what these numbers do Special thanks to: Philippe Antoine Tomas Korbar and Clang UndefinedBehaviorSanitizer OSS-Fuzz / ClusterFuzz ``` Signed-off-by: Aapo Talvensaari <[email protected]>
### Summary #### 2.6.0 ``` Release 2.6.0 Tue February 6 2024 Security fixes: #789 #814 CVE-2023-52425 -- Fix quadratic runtime issues with big tokens that can cause denial of service, in partial where dealing with compressed XML input. Applications that parsed a document in one go -- a single call to functions XML_Parse or XML_ParseBuffer -- were not affected. The smaller the chunks/buffers you use for parsing previously, the bigger the problem prior to the fix. Backporters should be careful to no omit parts of pull request #789 and to include earlier pull request #771, in order to not break the fix. #777 CVE-2023-52426 -- Fix billion laughs attacks for users compiling *without* XML_DTD defined (which is not common). Users with XML_DTD defined have been protected since Expat >=2.4.0 (and that was CVE-2013-0340 back then). Bug fixes: #753 Fix parse-size-dependent "invalid token" error for external entities that start with a byte order mark #780 Fix NULL pointer dereference in setContext via XML_ExternalEntityParserCreate for compilation with XML_DTD undefined #812 #813 Protect against closing entities out of order Other changes: #723 Improve support for arc4random/arc4random_buf #771 #788 Improve buffer growth in XML_GetBuffer and XML_Parse #761 #770 xmlwf: Support --help and --version #759 #770 xmlwf: Support custom buffer size for XML_GetBuffer and read #744 xmlwf: Improve language and URL clickability in help output #673 examples: Add new example "element_declarations.c" #764 Be stricter about macro XML_CONTEXT_BYTES at build time #765 Make inclusion to expat_config.h consistent #726 #727 Autotools: configure.ac: Support --disable-maintainer-mode #678 #705 .. #706 #733 #792 Autotools: Sync CMake templates with CMake 3.26 #795 Autotools: Make installation of shipped man page doc/xmlwf.1 independent of docbook2man availability #815 Autotools|CMake: Add missing -DXML_STATIC to pkg-config file section "Cflags.private" in order to fix compilation against static libexpat using pkg-config on Windows #724 #751 Autotools|CMake: Require a C99 compiler (a de-facto requirement already since Expat 2.2.2 of 2017) #793 Autotools|CMake: Fix PACKAGE_BUGREPORT variable #750 #786 Autotools|CMake: Make test suite require a C++11 compiler #749 CMake: Require CMake >=3.5.0 #672 CMake: Lowercase off_t and size_t to help a bug in Meson #746 CMake: Sort xmlwf sources alphabetically #785 CMake|Windows: Fix generation of DLL file version info #790 CMake: Build tests/benchmark/benchmark.c as well for a build with -DEXPAT_BUILD_TESTS=ON #745 #757 docs: Document the importance of isFinal + adjust tests accordingly #736 docs: Improve use of "NULL" and "null" #713 docs: Be specific about version of XML (XML 1.0r4) and version of C (C99); (XML 1.0r5 will need a sponsor.) #762 docs: reference.html: Promote function XML_ParseBuffer more #779 docs: reference.html: Add HTML anchors to XML_* macros #760 docs: reference.html: Upgrade to OK.css 1.2.0 #763 #739 docs: Fix typos #696 docs|CI: Use HTTPS URLs instead of HTTP at various places #669 #670 .. #692 #703 .. #733 #772 Address compiler warnings #798 #800 Address clang-tidy warnings #775 #776 Version info bumped from 9:10:8 (libexpat*.so.1.8.10) to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/ for what these numbers do Infrastructure: #700 #701 docs: Document security policy in file SECURITY.md #766 docs: Improve parse buffer variables in-code documentation #674 #738 .. #740 #747 .. #748 #781 #782 Refactor coverage and conformance tests #714 #716 Refactor debug level variables to unsigned long #671 Improve handling of empty environment variable value in function getDebugLevel (without visible user effect) #755 #774 .. #758 #783 .. #784 #787 tests: Improve test coverage with regard to parse chunk size #660 #797 #801 Fuzzing: Improve fuzzing coverage #367 #799 Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests #698 #721 CI: Resolve some Travis CI leftovers #669 CI: Be robust towards absence of Git tags #693 #694 CI: Set permissions to "contents: read" for security #709 CI: Pin all GitHub Actions to specific commits for security #739 CI: Reject spelling errors using codespell #798 CI: Enforce clang-tidy clean code #773 #808 .. #809 #810 CI: Upgrade Clang from 15 to 18 #796 CI: Start using Clang's Control Flow Integrity sanitizer #675 #720 #722 CI: Adapt to breaking changes in GitHub Actions Ubuntu images #689 CI: Adapt to breaking changes in Clang/LLVM Debian packaging #763 CI: Adapt to breaking changes in codespell #803 CI: Adapt to breaking changes in Cppcheck Special thanks to: Ivan Galkin Joyce Brum Philippe Antoine Rhodri James Snild Dolkow spookyahell Steven Garske and Clang AddressSanitizer Clang UndefinedBehaviorSanitizer codespell GCC Farm Project OSS-Fuzz Sony Mobile ``` #### 2.6.1 ``` Release 2.6.1 Thu February 29 2024 Bug fixes: #817 Make tests independent of CPU speed, and thus more robust #828 #836 Expose billion laughs API with XML_DTD defined and XML_GE undefined, regression from 2.6.0 Other changes: #829 Hide test-only code behind new internal macro #833 Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P #819 Address compiler warnings #832 #834 Version info bumped from 10:0:9 (libexpat*.so.1.9.0) to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/ for what these numbers do Infrastructure: #818 CI: Adapt to breaking changes in clang-format Special thanks to: David Hall Snild Dolkow ``` #### 2.6.2 ``` Release 2.6.2 Wed March 13 2024 Security fixes: #839 #842 CVE-2024-28757 -- Prevent billion laughs attacks with isolated use of external parsers. Please see the commit message of commit 1d50b80cf31de87750103656f6eb693746854aa8 for details. Bug fixes: #839 #841 Reject direct parameter entity recursion and avoid the related undefined behavior Other changes: #847 Autotools: Fix build for DOCBOOK_TO_MAN containing spaces #837 Add missing #821 and #824 to 2.6.1 change log #838 #843 Version info bumped from 10:1:9 (libexpat*.so.1.9.1) to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/ for what these numbers do Special thanks to: Philippe Antoine Tomas Korbar and Clang UndefinedBehaviorSanitizer OSS-Fuzz / ClusterFuzz ``` Signed-off-by: Aapo Talvensaari <[email protected]>
``` Release 2.6.0 Tue February 6 2024 Security fixes: #789 #814 CVE-2023-52425 -- Fix quadratic runtime issues with big tokens that can cause denial of service, in partial where dealing with compressed XML input. Applications that parsed a document in one go -- a single call to functions XML_Parse or XML_ParseBuffer -- were not affected. The smaller the chunks/buffers you use for parsing previously, the bigger the problem prior to the fix. Backporters should be careful to no omit parts of pull request #789 and to include earlier pull request #771, in order to not break the fix. #777 CVE-2023-52426 -- Fix billion laughs attacks for users compiling *without* XML_DTD defined (which is not common). Users with XML_DTD defined have been protected since Expat >=2.4.0 (and that was CVE-2013-0340 back then). Bug fixes: #753 Fix parse-size-dependent "invalid token" error for external entities that start with a byte order mark #780 Fix NULL pointer dereference in setContext via XML_ExternalEntityParserCreate for compilation with XML_DTD undefined #812 #813 Protect against closing entities out of order Other changes: #723 Improve support for arc4random/arc4random_buf #771 #788 Improve buffer growth in XML_GetBuffer and XML_Parse #761 #770 xmlwf: Support --help and --version #759 #770 xmlwf: Support custom buffer size for XML_GetBuffer and read #744 xmlwf: Improve language and URL clickability in help output #673 examples: Add new example "element_declarations.c" #764 Be stricter about macro XML_CONTEXT_BYTES at build time #765 Make inclusion to expat_config.h consistent #726 #727 Autotools: configure.ac: Support --disable-maintainer-mode #678 #705 .. #706 #733 #792 Autotools: Sync CMake templates with CMake 3.26 #795 Autotools: Make installation of shipped man page doc/xmlwf.1 independent of docbook2man availability #815 Autotools|CMake: Add missing -DXML_STATIC to pkg-config file section "Cflags.private" in order to fix compilation against static libexpat using pkg-config on Windows #724 #751 Autotools|CMake: Require a C99 compiler (a de-facto requirement already since Expat 2.2.2 of 2017) #793 Autotools|CMake: Fix PACKAGE_BUGREPORT variable #750 #786 Autotools|CMake: Make test suite require a C++11 compiler #749 CMake: Require CMake >=3.5.0 #672 CMake: Lowercase off_t and size_t to help a bug in Meson #746 CMake: Sort xmlwf sources alphabetically #785 CMake|Windows: Fix generation of DLL file version info #790 CMake: Build tests/benchmark/benchmark.c as well for a build with -DEXPAT_BUILD_TESTS=ON #745 #757 docs: Document the importance of isFinal + adjust tests accordingly #736 docs: Improve use of "NULL" and "null" #713 docs: Be specific about version of XML (XML 1.0r4) and version of C (C99); (XML 1.0r5 will need a sponsor.) #762 docs: reference.html: Promote function XML_ParseBuffer more #779 docs: reference.html: Add HTML anchors to XML_* macros #760 docs: reference.html: Upgrade to OK.css 1.2.0 #763 #739 docs: Fix typos #696 docs|CI: Use HTTPS URLs instead of HTTP at various places #669 #670 .. #692 #703 .. #733 #772 Address compiler warnings #798 #800 Address clang-tidy warnings #775 #776 Version info bumped from 9:10:8 (libexpat*.so.1.8.10) to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/ for what these numbers do Infrastructure: #700 #701 docs: Document security policy in file SECURITY.md #766 docs: Improve parse buffer variables in-code documentation #674 #738 .. #740 #747 .. #748 #781 #782 Refactor coverage and conformance tests #714 #716 Refactor debug level variables to unsigned long #671 Improve handling of empty environment variable value in function getDebugLevel (without visible user effect) #755 #774 .. #758 #783 .. #784 #787 tests: Improve test coverage with regard to parse chunk size #660 #797 #801 Fuzzing: Improve fuzzing coverage #367 #799 Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests #698 #721 CI: Resolve some Travis CI leftovers #669 CI: Be robust towards absence of Git tags #693 #694 CI: Set permissions to "contents: read" for security #709 CI: Pin all GitHub Actions to specific commits for security #739 CI: Reject spelling errors using codespell #798 CI: Enforce clang-tidy clean code #773 #808 .. #809 #810 CI: Upgrade Clang from 15 to 18 #796 CI: Start using Clang's Control Flow Integrity sanitizer #675 #720 #722 CI: Adapt to breaking changes in GitHub Actions Ubuntu images #689 CI: Adapt to breaking changes in Clang/LLVM Debian packaging #763 CI: Adapt to breaking changes in codespell #803 CI: Adapt to breaking changes in Cppcheck Special thanks to: Ivan Galkin Joyce Brum Philippe Antoine Rhodri James Snild Dolkow spookyahell Steven Garske and Clang AddressSanitizer Clang UndefinedBehaviorSanitizer codespell GCC Farm Project OSS-Fuzz Sony Mobile ``` ``` Release 2.6.1 Thu February 29 2024 Bug fixes: #817 Make tests independent of CPU speed, and thus more robust #828 #836 Expose billion laughs API with XML_DTD defined and XML_GE undefined, regression from 2.6.0 Other changes: #829 Hide test-only code behind new internal macro #833 Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P #819 Address compiler warnings #832 #834 Version info bumped from 10:0:9 (libexpat*.so.1.9.0) to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/ for what these numbers do Infrastructure: #818 CI: Adapt to breaking changes in clang-format Special thanks to: David Hall Snild Dolkow ``` ``` Release 2.6.2 Wed March 13 2024 Security fixes: #839 #842 CVE-2024-28757 -- Prevent billion laughs attacks with isolated use of external parsers. Please see the commit message of commit 1d50b80cf31de87750103656f6eb693746854aa8 for details. Bug fixes: #839 #841 Reject direct parameter entity recursion and avoid the related undefined behavior Other changes: #847 Autotools: Fix build for DOCBOOK_TO_MAN containing spaces #837 Add missing #821 and #824 to 2.6.1 change log #838 #843 Version info bumped from 10:1:9 (libexpat*.so.1.9.1) to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/ for what these numbers do Special thanks to: Philippe Antoine Tomas Korbar and Clang UndefinedBehaviorSanitizer OSS-Fuzz / ClusterFuzz ``` KAG-4331 Signed-off-by: Aapo Talvensaari <[email protected]>
I'm running Kong 0.5.3 on
c4.large
EC2 boxes, putting a reasonable amount of traffic (about 120 req/s per Kong machine) through it. CPU and memory usage seem pretty low, but I'm getting messages like the following in Kong's error log.Is this a sign that Kong is overloaded? If so, do I need to spin up more machines? Or is there some tuning I can do in
kong.yml
, e.g. to give more memory to nginx?There is loads of free memory available on the box:
The nginx processes look like this in
top
:The text was updated successfully, but these errors were encountered: