Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python projects that have Python 3.10+ syntax have broken code coverage #11419

Open
DavidKorczynski opened this issue Dec 29, 2023 · 6 comments
Open

Python projects that have Python 3.10+ syntax have broken code coverage #11419

DavidKorczynski opened this issue Dec 29, 2023 · 6 comments

Comments

@DavidKorczynski
Copy link
Collaborator

Ref: #11401 (comment)
@DavidKorczynski DavidKorczynski mentioned this issue Dec 29, 2023
Closed
@DavidKorczynski
Copy link
Collaborator Author

@apollo13 I gave this a go -- I think the right approach is to bump Python in the base-runner in this case, unless there is a concrete need for having older Python versions as well. I think this will likely be picked up and possibly landed early Jan.

@apollo13
Copy link
Contributor

apollo13 commented Dec 29, 2023 via email

@maflcko
Copy link
Contributor

maflcko commented Apr 2, 2024

See also #9532

DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Jun 4, 2024
@DaveLak
[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images
a9fc958 
The changes introduced here upgrade Python from 3.8 to 3.10.14 inside
the base-builder and base-runner images.

 ### base-builder changes:

Prior to these changes, base-builder compiled Python 3.8 from source
using sources downloaded from the official release servers at
https://www.python.org/ftp/python/. This updates the compiled version
to 3.10.14 (the latest 3.10 release) instead.

 ### base-runner changes:

Prior to these changes, base-runner installed Python 3.8 from the
default apt repository provided by the Ubuntu 20.04 image it's based
on. These apt repositories do not have a version of Python 3.10
available by default. This updates the base-runner to instead use a
multi-stage build to copy the same Python interpreter compiled by the
base-builder image into the runner image, which ensures both Python
versions remain in-sync while saving build time by re-using a pre-built
version.

 ## Motivation

- Code coverage does not work on Python projects that use Python 3.10+
  syntax, and will not work until this or similar changes are landed
  (see google#11419)
- Upgrading the base-image to use Ubuntu 22.04 (which provides more
  recent Python versions via apt) has been stated as being unlikely to
  happen any time soon (see google#3290)
- Many OSS-Fuzz integrated Python projects no longer support Python 3.8
  and have resorted to implementing ad-hoc workarounds to upgrade to
  newer Python versions, including installing Python from the Dead
  Snakes PPA.
  - This leads to fragmentation and hard to debug issues. Maintenance
    is easier when everyone is using the same version without issue.
- With [Python 3.8 reaching end of life soon (in 2024-10)][python-
  versions-EOL], it is likely that more Python projects will begin
  dropping support for 3.8, further increasing the number of broken
  builds and ad-hoc workarounds.
- Previous attempts at upgrading Python have stalled.

 ## Known & Expected Issues

Several project Dockerfiles and build scripts contain hard coded
references to python3.8 file system paths, and many more have implanted
ad-hoc workarounds to upgrade to newer Python versions than 3.8
(typically 3.9.) Additional changes are required to each of these
projects to ensure they successfully build after this upgrade to Python
3.10.

 ### Fuzz Introspector Caveat

Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is
not expected to introduce any new issues, it was not tested on these
changes and may require additional work.

 ## Possible Areas of Improvement

Using the base-builder image in a multi-stage build to copy the pre-
compiled Python into base-runner is effective, but feels like a
workaround that may be introducing tech debt. A cleaner approach would
be to extract the Python compilation into a discrete base image similar
to how `base-clang` works, and use that as the multi-stage builder in
images that need it.

---

Fixes:
- google#11419

Supersedes:
- google#9532
- google#11420

[python-versions-EOL]: https://devguide.python.org/versions/
DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Jun 4, 2024
@DaveLak
[django] Fix Coverage Build by Using Base Runner Provided Python
c724d61 
Replaces the ad-hoc workaround using the Dead Snakes PPA to install
Python 3.10, with the upgraded Python 3.10 version provided by
the base-builder and base-runner images, the latter of which resolves
the issue mentioned in google#11419.
Fixes: google#9638
@DaveLak DaveLak mentioned this issue Jun 4, 2024
Merged
@hugovk hugovk mentioned this issue Jul 16, 2024
Merged
@hugovk
Copy link
Contributor

hugovk commented Jul 16, 2024

Hello, it would be very helpful for us at Pillow because Python 3.8 is EOL in October which is when our next release is, and we've already removed support for 3.8 from the codebase.

But now OSS-Fuzz is failing for every PR, so we'll probably have to remove it from our CI until it's ready.

Re: python-pillow/Pillow#8183 (comment)

@radarhere radarhere mentioned this issue Jul 16, 2024
Merged
@radarhere radarhere mentioned this issue Aug 10, 2024
Merged
DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Aug 28, 2024
@DaveLak
[django] Fix Coverage Build by Using Base Runner Provided Python
185ac6b 
Replaces the ad-hoc workaround using the Dead Snakes PPA to install
Python 3.10, with the upgraded Python 3.10 version provided by
the base-builder and base-runner images, the latter of which resolves
the issue mentioned in google#11419.
Fixes: google#9638
This was referenced Sep 26, 2024
Closed
Merged
DavidKorczynski pushed a commit that referenced this issue Sep 30, 2024
@radarhere
[Pillow] Use Python 3.9 (#12546)
187d0f1 
Pillow's oss-fuzz is currently failing, because Pillow main has [dropped
support for Python
3.8](python-pillow/Pillow#8183).

Until a global solution is provided for #11419, this installs Python 3.9
to allow Pillow to work again.

cc @hugovk

Co-authored-by: Andrew Murray <[email protected]>
DaveLak added a commit to DaveLak/oss-fuzz that referenced this issue Nov 20, 2024
@DaveLak
[django] Fix Coverage Build by Using Base Runner Provided Python
8446d53 
Replaces the ad-hoc workaround using the Dead Snakes PPA to install
Python 3.10, with the upgraded Python 3.10 version provided by
the base-builder and base-runner images, the latter of which resolves
the issue mentioned in google#11419.
Fixes: google#9638
oliverchang added a commit that referenced this issue Nov 25, 2024
@DaveLak @oliverchang @radarhere
[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images (
93b417e 
#12027)

> [!NOTE]  
> I was looking for somewhere to get feedback from maintainers about
this approach to the Python 3.10 upgrade before attempting it, but the
discussion surrounding a Python upgrade has been rather fragmented
across many issues, PRs, and comment chains.
>
> For that reason, I felt it would be easier to propose with a working
example and dedicated PR.


#### Fixes:
- #11419
- #9638

#### Supersedes:
- #9532
- #11420


## Changes

The changes introduced here upgrade Python from 3.8 to 3.10.14 inside
the base-builder and base-runner images.

### Base Image Changes

| Image | Before Changes | After Changes |

|----------------|------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **base-builder** | Compiled Python 3.8 from source using official
release servers at https://www.python.org/ftp/python/. | Compiles Python
3.10.14 (the latest 3.10 release) from source using official release
servers at https://www.python.org/ftp/python/. |
| **base-runner** | Installed Python 3.8 from the default apt repository
provided by the Ubuntu 20.04 image. | Uses a multi-stage build to copy
the Python 3.10.14 interpreter compiled by the base-builder image,
ensuring version sync and saving build time by re-using a pre-built
version. |


## Known Impact on Projects

### 3.9 Workarounds That Can Be Removed

| Project    | Fix Link |
|------------|----------|
| dask |
DaveLak@417bbf5
|
| docutils |
DaveLak@e4c21ff
|
| dovecot |
DaveLak@7ab3ab6
|
| nbclassic |
DaveLak@5509b4e
|
| pandas |
DaveLak@0642a7a
|
| pybind11 |
DaveLak@a5bbdb3
|
| pyodbc |
DaveLak@afa2b5e
|
| qpid-proton|
DaveLak@f5bf756
|

### Anticipated Build Failures

#### Preexisting Failures 

##### Fix is Prepared

| Project               | Fix Link |
|-----------------------|----------|
| airflow |
DaveLak@60a0368
|
| ipython |
DaveLak@21ac68e
|
| networkx |
DaveLak@fc2f8c5
|
| numpy |
DaveLak@9383c87
|
| tensorflow-addons |
DaveLak@eed2bea
|
| django (coverage build)|
DaveLak@c724d61
|
| proto-plus-python |
DaveLak@37d973e
|
| dnspython | The upgraded pip version in the base-builder fixes the
currently failing build. |

##### Fix Requires Upstream Changes

| Project | Issue |
|---------|-------|
| pyvex | Currently failing on python 3.9 because `archinfo` dependency
requires >=3.10. Fails after the 3.10 upgrade because [the upstream
build script needs `python3.9` replaced with
`python3`](https://github.com/angr/pyvex/blob/f94c95636a3800c5bbd781ecf1e3fb0c0d9feec4/fuzzing/build.sh#L19-L23).
|

##### Requires More Investigation

| Project            | Issue |
|--------------------|-------|
| matplotlib | Upgrading Python & Pyinstaller does resolve the build
issues, but an error in the fuzz harness is exposed and must be resolved
for check_build to pass. The exception: `TypeError: Parser.non_math()
takes 2 positional arguments but 4 were given" in "File "fuzz_plt.py",
line 43, in TestOneInput`. |
| scipy | Upgrading Python & Pyinstaller does resolve the build issues,
but an error in the build step causes the build to fail. The error seems
related to the linking: "/usr/bin/ld: /usr/bin/ld: DWARF error: invalid
or unhandled FORM value: 0x25". When `export LDFLAGS="-fuse-ld=lld"` is
set, the error becomes: "`ld.lld: error: undefined symbol:
__asan_report_store4`". |
| pandas (Introspector only)| [This workaround in `build.sh` is the
issue](https://github.com/google/oss-fuzz/blob/1515519a665756d8a50a6c46abac8b431e5462ef/projects/pandas/build.sh#L22-L32).
|
| pycrypto | Failing with error: "`SystemError: PY_SSIZE_T_CLEAN macro
must be defined for '#' formats`". Seems like the issue described
[here](https://stackoverflow.com/a/71019907). Pycrypto is deprecated and
this is unlikely to be fixed upstream. |


## Possible Future Improvements

Using the base-builder image in a multi-stage build to copy the pre-
compiled Python into base-runner is effective, but feels like a
workaround that may be introducing tech debt. A cleaner approach would
be to extract the Python compilation into a discrete base image similar
to how `base-clang` works, and use that as the multi-stage builder in
images that need it.

### Fuzz Introspector Caveat

Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is
not expected to introduce any new issues, it was not tested on these
changes and may require additional work.

---

## Motivation

- Python [3.8 is reaching end of life in October
2024](https://devguide.python.org/versions/).
- The [Scientific Python Community already encourages dropping 3.8
support](https://scientific-python.org/specs/spec-0000/).
- This is evident when looking at which projects have resorted to
upgrading to newer Pythons using ad-hoc workarounds (see `numpy`,
`scipy`, `pandas`, etc.)
- It is likely that more Python projects will begin dropping support for
3.8, further increasing the number of broken builds and ad-hoc
workarounds.
- Code coverage does not work on Python projects that use Python 3.10+
syntax.
- Previous attempts at upgrading Python have stalled (see
google/clusterfuzz#3290 (comment)
& the issues linked under "Supersedes" above.)
- In recognition of the fact that OSS-Fuzz maintainers are stretched
thin, I thought I'd give it a shot.

---------

Co-authored-by: Oliver Chang <[email protected]>
Co-authored-by: Andrew Murray <[email protected]>
@radarhere
Copy link
Contributor

#12027 has now been merged.

@rgommers
Copy link

rgommers commented Dec 7, 2024 •
edited
Loading

Python 3.10 is still very old. The NumPy builds were failing for a long time, worked for 2 weeks after gh-12027, and are now failing again because NumPy bumped to python >=3.11.

OSS Fuzz is testing the main branch of a project. In this case, NumPy started a new dev cycle for 2.3.0 to be released in June 2025 (there are 2 feature releases per year, June and December). Python 3.10 is >3 years old now and will be almost 4 years old at that point. Running a Python version that old makes OSS Fuzz not usable for many Python projects. Can you please consider using a more recent Python version that works for all reasonably well-maintained packages? 3.11 or 3.12 are both good choices, 3.10 is not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

base-runner: bump python to 3.10 google/oss-fuzz
7 participants
@apollo13 @rgommers @DavidKorczynski @hugovk @radarhere @maflcko and others