Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lua VM crashed, reason: not enough memory #750

Closed
cb372 opened this issue Nov 27, 2015 · 12 comments
Closed

Lua VM crashed, reason: not enough memory #750

cb372 opened this issue Nov 27, 2015 · 12 comments

Comments

@cb372
Copy link

cb372 commented Nov 27, 2015

I'm running Kong 0.5.3 on c4.large EC2 boxes, putting a reasonable amount of traffic (about 120 req/s per Kong machine) through it. CPU and memory usage seem pretty low, but I'm getting messages like the following in Kong's error log.

Is this a sign that Kong is overloaded? If so, do I need to spin up more machines? Or is there some tuning I can do in kong.yml, e.g. to give more memory to nginx?

nginx: lua atpanic: Lua VM crashed, reason: not enough memory
*** stack smashing detected ***: nginx: worker process terminated
2015/11/27 15:47:41 [alert] 1205#0: worker process 23453 exited on signal 6 (core dumped)
nginx: lua atpanic: Lua VM crashed, reason: not enough memory
2015/11/27 16:15:14 [error] 28058#0: [lua] buffer.lua:53: [mashape-analytics] failed to create delayed batch sending timer: process exiting, context: ngx.timer, client: 54.216.32.247, server: 0.0.0.0:8000
2015/11/27 16:15:14 [error] 28058#0: [lua] init_worker.lua:12: create_timer(): [reports] failed to create timer: process exiting, context: ngx.timer
nginx: lua atpanic: Lua VM crashed, reason: not enough memory

There is loads of free memory available on the box:

ubuntu@ip-10-248-76-173:~$ free -h
             total       used       free     shared    buffers     cached
Mem:          3.7G       1.1G       2.5G       6.0M       102M       351M
-/+ buffers/cache:       718M       3.0G
Swap:           0B         0B         0B

The nginx processes look like this in top:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
29678 nobody    20   0  245516  61112   5032 S  11.3  1.6   0:22.35 nginx
28958 nobody    20   0  728236 542236   5340 S   3.0 14.1   3:02.60 nginx
@subnetmarco
Copy link
Member

Just a few questions to better understand the context:

  • What operating system are you running?
  • How did you install Kong (source/packages)?
  • What plugins did you add on the API?
  • Did you change the nginx configuration property in kong.yml?

@cb372
Copy link
Author

cb372 commented Nov 28, 2015

It's the Kong .deb package, running on Ubuntu Vivid. Nginx config is unchanged - the only thing I touched in kong.yml is the Cassandra hostnames.

I've added the key-auth and mashape-analytics plugins via the API. I'm also using rate limiting. There are a few thousand consumers registered, each with a rate limit and one or two keys.

Let me know if there's any more helpful info I can provide.

@thibaultcha
Copy link
Member

It is likely to me that the analytics plugin cannot buffer the amount of data at this pace and thus is causing the crash. This is similar to openresty/lua-nginx-module#148 and openresty/lua-nginx-module#230 which propose ways of investigating the issue.

For now:

  • are you performing requests with bodies and are those being logged by the analytics plugin? If so, they can take quite some space and that should be considered when configuring it.
  • how often are you emptying the buffers? conf.delay and conf.batch_size. Eventually consider lowering those values so the buffers are being flushed more often.
  • how many APIs are you hitting in your tests? The analytics plugin keeps 1 buffer/API/worker. If you are using, for example, 10 APIs, then 10 buffers are being used for each worker, and thus more resources are needed for them to perform.
  • do you see batches being sent from Kong to Galileo? Do you see logs in error.log saying so, and data in your Galileo dashboard?

I am not sure it is the reason but it is what seems more likely to me.

@thibaultcha
Copy link
Member

I am not sure this can be qualified as a bug since memory limitations are, granted, optimizable, but ultimately quite unavoidable. This needs more investigation (especially finding out if it can be optimized), before being qualified as so. The last question (do you see batches being sent from Kong to Galileo? Do you see logs in error.log saying so, and data in your Galileo dashboard?) is definitely a crucial point to know what is the cause of the problem here.

@subnetmarco
Copy link
Member

If we confirm the problem is the Galileo in-memory batch growing too big, this could be qualified as an enhancement for the plugin and a solution could be dumping the buffer on disk when it gets bigger than a specific size (kinda to how nginx buffers request/response bodies to file if they are too big).

@cb372
Copy link
Author

cb372 commented Nov 30, 2015

@thibaultcha My theory at the moment is, yes, it's the analytics plugin. I turned it off for a few hours today and didn't see a single error of this kind. I turned it on again just before I went home, so I'll check the logs again tomorrow and see if the errors came back.

are you performing requests with bodies and are those being logged by the analytics plugin?

No, body logging is disabled, and the requests are all GETs anyway

how often are you emptying the buffers? conf.delay and conf.batch_size

100 events and 2 seconds (i.e. the default values)

how many APIs are you hitting in your tests?

Just 1 API

do you see batches being sent from Kong to Galileo? Do you see logs in error.log saying so, and data in your Galileo dashboard?

I don't have Kong's debug logs enabled, so it's hard to see exactly when it is sending batches. The Galileo dashboard does show data, but it has huge gaps of an hour or so at a time.

If I'm reading the source of the analytics plugin right, the size of a single batch is limited but the queue of batches waiting to be sent is unbounded. This would explain the out-of-memory issue, if for any reason Kong was temporarily unable to connect to the Analytics server.

@thefosk 's suggestion of spooling to disk sounds good, but it also sounds quite complex and thus unlikely to be implemented for a while. As a first step I would be quite happy with:

  1. Make the batches queue bounded, with the max size configurable
  2. If the queue is full, just drop the batch on the floor and log a warning

I'm much happier losing analytics data for 100 events in a controlled manner than having Kong run out of memory, blow up without warning, and lose the data for the entire waiting-to-send queue.

@thibaultcha
Copy link
Member

Even if you don't have debug logs enabled, the error.log file will have errors logs when batches were not acknowledged by Galileo. It would be very useful here to know if any of those happen, and what is the status code returned by Galileo. Those informations are available in the logs. Currently, batches enqueued are dropped if Galileo is responding with a 400 status code, but it might have been changed on Galileo's side and other status codes could now be used, causing an invalid batch to not be dropped. This could be one reason.

Having a limit in the queue of batches is also a possibility, but it might not be needed if the error is coming from un-dropped invalid batches.

@cb372
Copy link
Author

cb372 commented Nov 30, 2015

I've just checked error.log again and found some errors coming from the plugin.

There are quite a few of these per day, but much fewer than the number of Lua VM crashed, reason: not enough memory logs.

2015/11/28 00:16:30 [error] 8894#0: [lua] buffer.lua:191: [mashape-analytics] socket server could not save the batch (100 ALFs 301424 bytes). Status: (504) Error: (), context: ngx.timer, client: 54.170.138.214, server: 0.0.0.0:8000
2015/11/28 00:54:33 [error] 8894#0: [lua] buffer.lua:191: [mashape-analytics] socket server could not save the batch (100 ALFs 313776 bytes). Status: (504) Error: (), context: ngx.timer, client: 54.216.32.247, server: 0.0.0.0:8000
2015/11/28 02:18:39 [error] 8894#0: [lua] buffer.lua:191: [mashape-analytics] socket server could not save the batch (100 ALFs 313838 bytes). Status: (504) Error: (), context: ngx.timer, client: 54.78.132.159, server: 0.0.0.0:8000

These also appeared, but only 8 of them over one 5 minute period:

2015/11/30 07:43:54 [error] 14338#0: [lua] buffer.lua:191: [mashape-analytics] socket server could not save the batch (100 ALFs 303971 bytes). Status: (207) Error: (Valid ALFs: 0/100 saved
), context: ngx.timer, client: 54.78.132.159, server: 0.0.0.0:8000
2015/11/30 07:44:10 [error] 14338#0: [lua] buffer.lua:191: [mashape-analytics] socket server could not save the batch (100 ALFs 299382 bytes). Status: (207) Error: (Valid ALFs: 0/100 saved
), context: ngx.timer, client: 54.216.251.254, server: 0.0.0.0:8000
2015/11/30 07:44:41 [error] 14338#0: [lua] buffer.lua:191: [mashape-analytics] socket server could not save the batch (100 ALFs 304055 bytes). Status: (207) Error: (Valid ALFs: 0/100 saved
), context: ngx.timer, client: 54.155.39.94, server: 0.0.0.0:8000

Is it possible that the Analytics server is not returning (many) error responses but is instead taking a long time to respond to the requests? That would be consistent with the behaviour we are seeing, i.e. out-of-memory errors + not many error logs. The 504 responses being returned also suggest that the Analytics server might be struggling to handle the load.

@cb372
Copy link
Author

cb372 commented Dec 1, 2015

Sorry to repeat myself, but I strongly recommend having a bound on that queue. Relying on the HTTP response code is not enough - if the analytics server responds slowly (as I suspect it is doing), or if it goes down for a few minutes, as it inevitably will now and again, then the queue will start to grow.

It's a simple matter of fault isolation: auxiliary functionality such as the analytics plugin should not be able to cause any adverse effect on the main functionality of Kong, no matter what happens. And an out-of-memory error crashing the whole process is the mother of all adverse effects :)

I would really appreciate a quick fix for this. I was planning to put Kong into production this week, but I won't be able to do that, at least not with the analytics plugin enabled, until this issue is fixed. And since we are paying for Galileo, disabling the analytics would be a real shame.

@thibaultcha
Copy link
Member

I said "it might" not be needed, in the sense that the issue could have not existed if Kong was handling 207. However, yes, it was already my intention to implement a cap on that queue after your first suggestion.

I have done so, as well as other improvements for the buffer in #757, which is opened against master as a hotfix. I am hoping to finish testing it soon, so we can release a new version (0.5.4) in the next day(s).

thibaultcha added a commit that referenced this issue Dec 2, 2015
As reported by #750, the buffer is vulnerable to a heavy load because of
its `sending_queue` of batches pending for sending. It also does not
handle the new 207 HTTP status code returned by the collector in case of
invalid ALFs.

Changes:
- handle 207 HTTP status code by discarding the batch. Some ALFs in it
  will have been saved, and the invalid one(s) should not be retried.
- implement a maximum size (in MB) for the sending_queue, as suggested
  by #750. Originally, I was about to implement such a size limitation
  by "number of batches pending", but it would not be intuitive for
  users to know what fits best their use case, because ALF sizes varies
  from one API to another, and one endpoint to another. By defining it
  in MB it is easier for users to chose a value. The default value of
  10MB has been chosen after performing some benchmarking, and should
  handle from 300 to 500 req/s depending on the ALFs sizes.
  When the `sending_queue` has reached its limit, the current ALFs in
  the buffer will be **discarded**.
- implement a retry policy. Instead of insisting on retrying to send batches
  when the collector cannot be reached, a delay is computed an
  exponentially increases on each failure to connect to the collector.
  That delay is shared by all workers. This avoids to load the collector
  when it is having difficulties and saves up bandwidth on Kong's side.
  As soon as the collector can be reached again, the delay is reset.
  Currently, the minimum retry delay is 1s and the maximum is 60s. Those
  values cannot be configured.
- no more line jumps in logs printing responses from the collector.
@cb372
Copy link
Author

cb372 commented Dec 2, 2015

Awesome, thanks for the quick response!

@thibaultcha
Copy link
Member

I believe we can close this now that #757 is merged and since 0.5.4 is coming (#767). Thanks for reporting.

kikito pushed a commit that referenced this issue Apr 23, 2024
### Summary

#### 2.6.0
```
Release 2.6.0 Tue February 6 2024
        Security fixes:
      #789 #814  CVE-2023-52425 -- Fix quadratic runtime issues with big tokens
                   that can cause denial of service, in partial where
                   dealing with compressed XML input.  Applications
                   that parsed a document in one go -- a single call to
                   functions XML_Parse or XML_ParseBuffer -- were not affected.
                   The smaller the chunks/buffers you use for parsing
                   previously, the bigger the problem prior to the fix.
                   Backporters should be careful to no omit parts of
                   pull request #789 and to include earlier pull request #771,
                   in order to not break the fix.
           #777  CVE-2023-52426 -- Fix billion laughs attacks for users
                   compiling *without* XML_DTD defined (which is not common).
                   Users with XML_DTD defined have been protected since
                   Expat >=2.4.0 (and that was CVE-2013-0340 back then).

        Bug fixes:
            #753  Fix parse-size-dependent "invalid token" error for
                    external entities that start with a byte order mark
            #780  Fix NULL pointer dereference in setContext via
                    XML_ExternalEntityParserCreate for compilation with
                    XML_DTD undefined
       #812 #813  Protect against closing entities out of order

        Other changes:
            #723  Improve support for arc4random/arc4random_buf
       #771 #788  Improve buffer growth in XML_GetBuffer and XML_Parse
       #761 #770  xmlwf: Support --help and --version
       #759 #770  xmlwf: Support custom buffer size for XML_GetBuffer and read
            #744  xmlwf: Improve language and URL clickability in help output
            #673  examples: Add new example "element_declarations.c"
            #764  Be stricter about macro XML_CONTEXT_BYTES at build time
            #765  Make inclusion to expat_config.h consistent
       #726 #727  Autotools: configure.ac: Support --disable-maintainer-mode
    #678 #705 ..
  #706 #733 #792  Autotools: Sync CMake templates with CMake 3.26
            #795  Autotools: Make installation of shipped man page doc/xmlwf.1
                    independent of docbook2man availability
            #815  Autotools|CMake: Add missing -DXML_STATIC to pkg-config file
                    section "Cflags.private" in order to fix compilation
                    against static libexpat using pkg-config on Windows
       #724 #751  Autotools|CMake: Require a C99 compiler
                    (a de-facto requirement already since Expat 2.2.2 of 2017)
            #793  Autotools|CMake: Fix PACKAGE_BUGREPORT variable
       #750 #786  Autotools|CMake: Make test suite require a C++11 compiler
            #749  CMake: Require CMake >=3.5.0
            #672  CMake: Lowercase off_t and size_t to help a bug in Meson
            #746  CMake: Sort xmlwf sources alphabetically
            #785  CMake|Windows: Fix generation of DLL file version info
            #790  CMake: Build tests/benchmark/benchmark.c as well for
                    a build with -DEXPAT_BUILD_TESTS=ON
       #745 #757  docs: Document the importance of isFinal + adjust tests
                    accordingly
            #736  docs: Improve use of "NULL" and "null"
            #713  docs: Be specific about version of XML (XML 1.0r4)
                    and version of C (C99); (XML 1.0r5 will need a sponsor.)
            #762  docs: reference.html: Promote function XML_ParseBuffer more
            #779  docs: reference.html: Add HTML anchors to XML_* macros
            #760  docs: reference.html: Upgrade to OK.css 1.2.0
       #763 #739  docs: Fix typos
            #696  docs|CI: Use HTTPS URLs instead of HTTP at various places
    #669 #670 ..
    #692 #703 ..
       #733 #772  Address compiler warnings
       #798 #800  Address clang-tidy warnings
       #775 #776  Version info bumped from 9:10:8 (libexpat*.so.1.8.10)
                    to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
       #700 #701  docs: Document security policy in file SECURITY.md
            #766  docs: Improve parse buffer variables in-code documentation
    #674 #738 ..
    #740 #747 ..
  #748 #781 #782  Refactor coverage and conformance tests
       #714 #716  Refactor debug level variables to unsigned long
            #671  Improve handling of empty environment variable value
                    in function getDebugLevel (without visible user effect)
    #755 #774 ..
    #758 #783 ..
       #784 #787  tests: Improve test coverage with regard to parse chunk size
  #660 #797 #801  Fuzzing: Improve fuzzing coverage
       #367 #799  Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests
       #698 #721  CI: Resolve some Travis CI leftovers
            #669  CI: Be robust towards absence of Git tags
       #693 #694  CI: Set permissions to "contents: read" for security
            #709  CI: Pin all GitHub Actions to specific commits for security
            #739  CI: Reject spelling errors using codespell
            #798  CI: Enforce clang-tidy clean code
    #773 #808 ..
       #809 #810  CI: Upgrade Clang from 15 to 18
            #796  CI: Start using Clang's Control Flow Integrity sanitizer
  #675 #720 #722  CI: Adapt to breaking changes in GitHub Actions Ubuntu images
            #689  CI: Adapt to breaking changes in Clang/LLVM Debian packaging
            #763  CI: Adapt to breaking changes in codespell
            #803  CI: Adapt to breaking changes in Cppcheck

        Special thanks to:
            Ivan Galkin
            Joyce Brum
            Philippe Antoine
            Rhodri James
            Snild Dolkow
            spookyahell
            Steven Garske
                 and
            Clang AddressSanitizer
            Clang UndefinedBehaviorSanitizer
            codespell
            GCC Farm Project
            OSS-Fuzz
            Sony Mobile
```

#### 2.6.1
```
Release 2.6.1 Thu February 29 2024
        Bug fixes:
            #817  Make tests independent of CPU speed, and thus more robust
       #828 #836  Expose billion laughs API with XML_DTD defined and
                    XML_GE undefined, regression from 2.6.0

        Other changes:
            #829  Hide test-only code behind new internal macro
            #833  Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P
            #819  Address compiler warnings
       #832 #834  Version info bumped from 10:0:9 (libexpat*.so.1.9.0)
                    to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
            #818  CI: Adapt to breaking changes in clang-format

        Special thanks to:
            David Hall
            Snild Dolkow
```

#### 2.6.2
```
Release 2.6.2 Wed March 13 2024
        Security fixes:
       #839 #842  CVE-2024-28757 -- Prevent billion laughs attacks with
                    isolated use of external parsers.  Please see the commit
                    message of commit 1d50b80cf31de87750103656f6eb693746854aa8
                    for details.

        Bug fixes:
       #839 #841  Reject direct parameter entity recursion
                    and avoid the related undefined behavior

        Other changes:
            #847  Autotools: Fix build for DOCBOOK_TO_MAN containing spaces
            #837  Add missing #821 and #824 to 2.6.1 change log
       #838 #843  Version info bumped from 10:1:9 (libexpat*.so.1.9.1)
                    to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/
                    for what these numbers do

        Special thanks to:
            Philippe Antoine
            Tomas Korbar
                 and
            Clang UndefinedBehaviorSanitizer
            OSS-Fuzz / ClusterFuzz
```

Signed-off-by: Aapo Talvensaari <[email protected]>
bungle added a commit that referenced this issue Apr 23, 2024
### Summary

#### 2.6.0
```
Release 2.6.0 Tue February 6 2024
        Security fixes:
      #789 #814  CVE-2023-52425 -- Fix quadratic runtime issues with big tokens
                   that can cause denial of service, in partial where
                   dealing with compressed XML input.  Applications
                   that parsed a document in one go -- a single call to
                   functions XML_Parse or XML_ParseBuffer -- were not affected.
                   The smaller the chunks/buffers you use for parsing
                   previously, the bigger the problem prior to the fix.
                   Backporters should be careful to no omit parts of
                   pull request #789 and to include earlier pull request #771,
                   in order to not break the fix.
           #777  CVE-2023-52426 -- Fix billion laughs attacks for users
                   compiling *without* XML_DTD defined (which is not common).
                   Users with XML_DTD defined have been protected since
                   Expat >=2.4.0 (and that was CVE-2013-0340 back then).

        Bug fixes:
            #753  Fix parse-size-dependent "invalid token" error for
                    external entities that start with a byte order mark
            #780  Fix NULL pointer dereference in setContext via
                    XML_ExternalEntityParserCreate for compilation with
                    XML_DTD undefined
       #812 #813  Protect against closing entities out of order

        Other changes:
            #723  Improve support for arc4random/arc4random_buf
       #771 #788  Improve buffer growth in XML_GetBuffer and XML_Parse
       #761 #770  xmlwf: Support --help and --version
       #759 #770  xmlwf: Support custom buffer size for XML_GetBuffer and read
            #744  xmlwf: Improve language and URL clickability in help output
            #673  examples: Add new example "element_declarations.c"
            #764  Be stricter about macro XML_CONTEXT_BYTES at build time
            #765  Make inclusion to expat_config.h consistent
       #726 #727  Autotools: configure.ac: Support --disable-maintainer-mode
    #678 #705 ..
  #706 #733 #792  Autotools: Sync CMake templates with CMake 3.26
            #795  Autotools: Make installation of shipped man page doc/xmlwf.1
                    independent of docbook2man availability
            #815  Autotools|CMake: Add missing -DXML_STATIC to pkg-config file
                    section "Cflags.private" in order to fix compilation
                    against static libexpat using pkg-config on Windows
       #724 #751  Autotools|CMake: Require a C99 compiler
                    (a de-facto requirement already since Expat 2.2.2 of 2017)
            #793  Autotools|CMake: Fix PACKAGE_BUGREPORT variable
       #750 #786  Autotools|CMake: Make test suite require a C++11 compiler
            #749  CMake: Require CMake >=3.5.0
            #672  CMake: Lowercase off_t and size_t to help a bug in Meson
            #746  CMake: Sort xmlwf sources alphabetically
            #785  CMake|Windows: Fix generation of DLL file version info
            #790  CMake: Build tests/benchmark/benchmark.c as well for
                    a build with -DEXPAT_BUILD_TESTS=ON
       #745 #757  docs: Document the importance of isFinal + adjust tests
                    accordingly
            #736  docs: Improve use of "NULL" and "null"
            #713  docs: Be specific about version of XML (XML 1.0r4)
                    and version of C (C99); (XML 1.0r5 will need a sponsor.)
            #762  docs: reference.html: Promote function XML_ParseBuffer more
            #779  docs: reference.html: Add HTML anchors to XML_* macros
            #760  docs: reference.html: Upgrade to OK.css 1.2.0
       #763 #739  docs: Fix typos
            #696  docs|CI: Use HTTPS URLs instead of HTTP at various places
    #669 #670 ..
    #692 #703 ..
       #733 #772  Address compiler warnings
       #798 #800  Address clang-tidy warnings
       #775 #776  Version info bumped from 9:10:8 (libexpat*.so.1.8.10)
                    to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
       #700 #701  docs: Document security policy in file SECURITY.md
            #766  docs: Improve parse buffer variables in-code documentation
    #674 #738 ..
    #740 #747 ..
  #748 #781 #782  Refactor coverage and conformance tests
       #714 #716  Refactor debug level variables to unsigned long
            #671  Improve handling of empty environment variable value
                    in function getDebugLevel (without visible user effect)
    #755 #774 ..
    #758 #783 ..
       #784 #787  tests: Improve test coverage with regard to parse chunk size
  #660 #797 #801  Fuzzing: Improve fuzzing coverage
       #367 #799  Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests
       #698 #721  CI: Resolve some Travis CI leftovers
            #669  CI: Be robust towards absence of Git tags
       #693 #694  CI: Set permissions to "contents: read" for security
            #709  CI: Pin all GitHub Actions to specific commits for security
            #739  CI: Reject spelling errors using codespell
            #798  CI: Enforce clang-tidy clean code
    #773 #808 ..
       #809 #810  CI: Upgrade Clang from 15 to 18
            #796  CI: Start using Clang's Control Flow Integrity sanitizer
  #675 #720 #722  CI: Adapt to breaking changes in GitHub Actions Ubuntu images
            #689  CI: Adapt to breaking changes in Clang/LLVM Debian packaging
            #763  CI: Adapt to breaking changes in codespell
            #803  CI: Adapt to breaking changes in Cppcheck

        Special thanks to:
            Ivan Galkin
            Joyce Brum
            Philippe Antoine
            Rhodri James
            Snild Dolkow
            spookyahell
            Steven Garske
                 and
            Clang AddressSanitizer
            Clang UndefinedBehaviorSanitizer
            codespell
            GCC Farm Project
            OSS-Fuzz
            Sony Mobile
```

#### 2.6.1
```
Release 2.6.1 Thu February 29 2024
        Bug fixes:
            #817  Make tests independent of CPU speed, and thus more robust
       #828 #836  Expose billion laughs API with XML_DTD defined and
                    XML_GE undefined, regression from 2.6.0

        Other changes:
            #829  Hide test-only code behind new internal macro
            #833  Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P
            #819  Address compiler warnings
       #832 #834  Version info bumped from 10:0:9 (libexpat*.so.1.9.0)
                    to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
            #818  CI: Adapt to breaking changes in clang-format

        Special thanks to:
            David Hall
            Snild Dolkow
```

#### 2.6.2
```
Release 2.6.2 Wed March 13 2024
        Security fixes:
       #839 #842  CVE-2024-28757 -- Prevent billion laughs attacks with
                    isolated use of external parsers.  Please see the commit
                    message of commit 1d50b80cf31de87750103656f6eb693746854aa8
                    for details.

        Bug fixes:
       #839 #841  Reject direct parameter entity recursion
                    and avoid the related undefined behavior

        Other changes:
            #847  Autotools: Fix build for DOCBOOK_TO_MAN containing spaces
            #837  Add missing #821 and #824 to 2.6.1 change log
       #838 #843  Version info bumped from 10:1:9 (libexpat*.so.1.9.1)
                    to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/
                    for what these numbers do

        Special thanks to:
            Philippe Antoine
            Tomas Korbar
                 and
            Clang UndefinedBehaviorSanitizer
            OSS-Fuzz / ClusterFuzz
```

Signed-off-by: Aapo Talvensaari <[email protected]>
tysoekong pushed a commit that referenced this issue Apr 26, 2024
```
Release 2.6.0 Tue February 6 2024
        Security fixes:
      #789 #814  CVE-2023-52425 -- Fix quadratic runtime issues with big tokens
                   that can cause denial of service, in partial where
                   dealing with compressed XML input.  Applications
                   that parsed a document in one go -- a single call to
                   functions XML_Parse or XML_ParseBuffer -- were not affected.
                   The smaller the chunks/buffers you use for parsing
                   previously, the bigger the problem prior to the fix.
                   Backporters should be careful to no omit parts of
                   pull request #789 and to include earlier pull request #771,
                   in order to not break the fix.
           #777  CVE-2023-52426 -- Fix billion laughs attacks for users
                   compiling *without* XML_DTD defined (which is not common).
                   Users with XML_DTD defined have been protected since
                   Expat >=2.4.0 (and that was CVE-2013-0340 back then).

        Bug fixes:
            #753  Fix parse-size-dependent "invalid token" error for
                    external entities that start with a byte order mark
            #780  Fix NULL pointer dereference in setContext via
                    XML_ExternalEntityParserCreate for compilation with
                    XML_DTD undefined
       #812 #813  Protect against closing entities out of order

        Other changes:
            #723  Improve support for arc4random/arc4random_buf
       #771 #788  Improve buffer growth in XML_GetBuffer and XML_Parse
       #761 #770  xmlwf: Support --help and --version
       #759 #770  xmlwf: Support custom buffer size for XML_GetBuffer and read
            #744  xmlwf: Improve language and URL clickability in help output
            #673  examples: Add new example "element_declarations.c"
            #764  Be stricter about macro XML_CONTEXT_BYTES at build time
            #765  Make inclusion to expat_config.h consistent
       #726 #727  Autotools: configure.ac: Support --disable-maintainer-mode
    #678 #705 ..
  #706 #733 #792  Autotools: Sync CMake templates with CMake 3.26
            #795  Autotools: Make installation of shipped man page doc/xmlwf.1
                    independent of docbook2man availability
            #815  Autotools|CMake: Add missing -DXML_STATIC to pkg-config file
                    section "Cflags.private" in order to fix compilation
                    against static libexpat using pkg-config on Windows
       #724 #751  Autotools|CMake: Require a C99 compiler
                    (a de-facto requirement already since Expat 2.2.2 of 2017)
            #793  Autotools|CMake: Fix PACKAGE_BUGREPORT variable
       #750 #786  Autotools|CMake: Make test suite require a C++11 compiler
            #749  CMake: Require CMake >=3.5.0
            #672  CMake: Lowercase off_t and size_t to help a bug in Meson
            #746  CMake: Sort xmlwf sources alphabetically
            #785  CMake|Windows: Fix generation of DLL file version info
            #790  CMake: Build tests/benchmark/benchmark.c as well for
                    a build with -DEXPAT_BUILD_TESTS=ON
       #745 #757  docs: Document the importance of isFinal + adjust tests
                    accordingly
            #736  docs: Improve use of "NULL" and "null"
            #713  docs: Be specific about version of XML (XML 1.0r4)
                    and version of C (C99); (XML 1.0r5 will need a sponsor.)
            #762  docs: reference.html: Promote function XML_ParseBuffer more
            #779  docs: reference.html: Add HTML anchors to XML_* macros
            #760  docs: reference.html: Upgrade to OK.css 1.2.0
       #763 #739  docs: Fix typos
            #696  docs|CI: Use HTTPS URLs instead of HTTP at various places
    #669 #670 ..
    #692 #703 ..
       #733 #772  Address compiler warnings
       #798 #800  Address clang-tidy warnings
       #775 #776  Version info bumped from 9:10:8 (libexpat*.so.1.8.10)
                    to 10:0:9 (libexpat*.so.1.9.0); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
       #700 #701  docs: Document security policy in file SECURITY.md
            #766  docs: Improve parse buffer variables in-code documentation
    #674 #738 ..
    #740 #747 ..
  #748 #781 #782  Refactor coverage and conformance tests
       #714 #716  Refactor debug level variables to unsigned long
            #671  Improve handling of empty environment variable value
                    in function getDebugLevel (without visible user effect)
    #755 #774 ..
    #758 #783 ..
       #784 #787  tests: Improve test coverage with regard to parse chunk size
  #660 #797 #801  Fuzzing: Improve fuzzing coverage
       #367 #799  Fuzzing|CI: Start running OSS-Fuzz fuzzing regression tests
       #698 #721  CI: Resolve some Travis CI leftovers
            #669  CI: Be robust towards absence of Git tags
       #693 #694  CI: Set permissions to "contents: read" for security
            #709  CI: Pin all GitHub Actions to specific commits for security
            #739  CI: Reject spelling errors using codespell
            #798  CI: Enforce clang-tidy clean code
    #773 #808 ..
       #809 #810  CI: Upgrade Clang from 15 to 18
            #796  CI: Start using Clang's Control Flow Integrity sanitizer
  #675 #720 #722  CI: Adapt to breaking changes in GitHub Actions Ubuntu images
            #689  CI: Adapt to breaking changes in Clang/LLVM Debian packaging
            #763  CI: Adapt to breaking changes in codespell
            #803  CI: Adapt to breaking changes in Cppcheck

        Special thanks to:
            Ivan Galkin
            Joyce Brum
            Philippe Antoine
            Rhodri James
            Snild Dolkow
            spookyahell
            Steven Garske
                 and
            Clang AddressSanitizer
            Clang UndefinedBehaviorSanitizer
            codespell
            GCC Farm Project
            OSS-Fuzz
            Sony Mobile
```

```
Release 2.6.1 Thu February 29 2024
        Bug fixes:
            #817  Make tests independent of CPU speed, and thus more robust
       #828 #836  Expose billion laughs API with XML_DTD defined and
                    XML_GE undefined, regression from 2.6.0

        Other changes:
            #829  Hide test-only code behind new internal macro
            #833  Autotools: Reject expat_config.h.in defining SIZEOF_VOID_P
            #819  Address compiler warnings
       #832 #834  Version info bumped from 10:0:9 (libexpat*.so.1.9.0)
                    to 10:1:9 (libexpat*.so.1.9.1); see https://verbump.de/
                    for what these numbers do

        Infrastructure:
            #818  CI: Adapt to breaking changes in clang-format

        Special thanks to:
            David Hall
            Snild Dolkow
```

```
Release 2.6.2 Wed March 13 2024
        Security fixes:
       #839 #842  CVE-2024-28757 -- Prevent billion laughs attacks with
                    isolated use of external parsers.  Please see the commit
                    message of commit 1d50b80cf31de87750103656f6eb693746854aa8
                    for details.

        Bug fixes:
       #839 #841  Reject direct parameter entity recursion
                    and avoid the related undefined behavior

        Other changes:
            #847  Autotools: Fix build for DOCBOOK_TO_MAN containing spaces
            #837  Add missing #821 and #824 to 2.6.1 change log
       #838 #843  Version info bumped from 10:1:9 (libexpat*.so.1.9.1)
                    to 10:2:9 (libexpat*.so.1.9.2); see https://verbump.de/
                    for what these numbers do

        Special thanks to:
            Philippe Antoine
            Tomas Korbar
                 and
            Clang UndefinedBehaviorSanitizer
            OSS-Fuzz / ClusterFuzz
```

KAG-4331

Signed-off-by: Aapo Talvensaari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants