Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-115133: test_xml_etree.py: Fix for Expat >=2.6.0 with reparse deferral (fixes #115133) #115138

Conversation

hartwork
Copy link
Contributor

@hartwork hartwork commented Feb 7, 2024

@hartwork hartwork force-pushed the issue-115133-fix-etree-xml-pull-parser-tests-for-expat-2-6-0 branch from 3c711d7 to a147a3e Compare February 7, 2024 14:50
@serhiy-storchaka
Copy link
Member

It rather defeats the purpose of the test.

The purpose of the incremental parsing is that you can process the XML file without reading all data and building the full tree. This test tests that you get the parsed data as fast as the corresponding input data is available. The modified test only tests that it is available after feeding all input data and closing the parser. But you do not need the incremental parsing for this.

@mgorny
Copy link
Contributor

mgorny commented Feb 7, 2024

I wonder how this affects stuff like XMPP where you actually need incremental parsing for things to work at all.

@Snild-Sony
Copy link

This test tests that you get the parsed data as fast as the corresponding input data is available. The modified test only tests that it is available after feeding all input data and closing the parser. But you do not need the incremental parsing for this.

I'm not sure how to guarantee that one gets the data "as fast as the input", but it should at least be possible to assert that some parsing happens before close(). Would that be close enough?

I wonder how this affects stuff like XMPP where you actually need incremental parsing for things to work at all.

Probably not well, at least in general. With XMPP specifically, you may get lucky and always send a whole packet for parsing at once (or just generally have small tokens), in which case there will be no incomplete tokens and therefore no need for reparse deferral. But it will be very implementation-dependent.

For such a special usecase, this should work as a "flush" operation:

XML_SetReparseDeferralEnabled(parser, XML_FALSE);
XML_Parse(parser, NULL, 0, XML_FALSE); // or the last data of your "batch", if you know that ahead of time
XML_SetReparseDeferralEnabled(parser, XML_TRUE);

@hardfalcon
Copy link

hardfalcon commented Feb 7, 2024

In case it's relevant to anyone: Python 3.11.8, built with the patch from this pull request against expat 2.6.0, and also using expat 2.6.0 after installing said Python package, is working just fine for me, both for running Gajim (an XMPP client written in Python) and for running a matrix-synapse server.

@mweinelt mweinelt mentioned this pull request Feb 10, 2024
13 tasks
@serhiy-storchaka
Copy link
Member

Thank you for your PR, but it was fixed by #115164 which keeps some of more strict testing.

@hartwork hartwork deleted the issue-115133-fix-etree-xml-pull-parser-tests-for-expat-2-6-0 branch February 17, 2024 13:54
gpshead pushed a commit that referenced this pull request Feb 29, 2024
…GH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 2, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this pull request Mar 4, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
gpshead pushed a commit that referenced this pull request Mar 6, 2024
…-52425) (GH-115623) (GH-116248)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

(cherry picked from commit 6a95676)
(cherry picked from commit 73807eb)
(cherry picked from commit eda2963)

---------

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <[email protected]>
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 6, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
github-actions bot pushed a commit to m-aciek/python-docs-weblate that referenced this pull request Mar 6, 2024
…-52425) (GH-115623) (GH-116248)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python/cpython#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

(cherry picked from commit 6a95676bb526261434dd068d6c49927c44d24a9b)
(cherry picked from commit 73807eb634315f70a464a18feaae33d9e065de09)
(cherry picked from commit eda2963378a3c292cf6bb202bb00e94e46ee6d90)

---------

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <[email protected]>

CPython-sync-commit-latest: 0a01ed6c2a116bd3e174fce33c21d84d650de569
gpshead pushed a commit that referenced this pull request Mar 6, 2024
…-52425) (GH-115623) (#116268)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

(cherry picked from commit 6a95676)
(cherry picked from commit 73807eb)
(cherry picked from commit eda2963)

---------

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <[email protected]>
ambv pushed a commit that referenced this pull request Mar 6, 2024
…-52425) (GH-115623) (GH-116270)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

Co-authored-by: Gregory P. Smith <[email protected]>
ambv pushed a commit that referenced this pull request Mar 6, 2024
…52425) (GH-115623) (GH-116272)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

Co-authored-by: Gregory P. Smith <[email protected]>
ambv pushed a commit that referenced this pull request Mar 6, 2024
…52425) (GH-115623) (GH-116275)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

Co-authored-by: Gregory P. Smith <[email protected]>
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
jessecomeau87 pushed a commit to jessecomeau87/Python that referenced this pull request May 20, 2024
… (GH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python/cpython#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting review tests Tests in the Lib/test dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test.test_xml_etree*.XMLPullParserTest.test_simple_xml fails with (system) expat 2.6.0
5 participants