Connection: Close not set when needed - breaks downloads.

[tomsommer]

#657 breaks downloads towards https://api.wordpress.org/ and many other sites when using Curl 7.29.0 (and perhaps other versions)

Error: RuntimeException: Failed to get url 'https://api.wordpress.org/core/version-check/1.7/?locale=en_US': cURL error 28: Operation timed out after 10000 milliseconds with 807 out of -1 bytes received.

It also causes issues with the REST API in Site Health with the error: REST API response: (http_request_failed) cURL error 28: Operation timed out after 10005 milliseconds with XXX out of XXX bytes received”

It also prevents WordPress plugin and core updates, basically anything that relies on the internal Curl handler in WordPress.

Adding back Connection: Close solves the issue. Please revert #657 ASAP and push the fix to WordPress 6.4

If you are just looking to fix the cURL timeouts issue in WordPress 6.4 Read here:

To fix the issue, replace /wp-includes/Requests/src/Transport/Curl.php with the following file: Curl.php.zip or see #838 (comment)

UPDATE: No need to do any manual fix, updating the WordPress Core to release 6.4.1 should work just fine now and resolve the issue.

[jrfnl]

@tomsommer PR #657 fixes a bug. Reverting it recreates the bug. What makes this bug more important than the previous bug ?

I suggest someone do some research as to why the Curl version which is checked for now (7.22) is not correct/causing problems, if another version should be checked for instead (or a Curl constant) and iterate on the previous fix.

[tomsommer]

@jrfnl I agree with that, I suggest the author of #657 should account for that (or the person who merged this)? Right now the change breaks previously working versions of Curl, versions which are bundled with production PHP versions (Remi PHP) and Operating Systems (RHEL7/CentOS 7), which makes this bug more important - IMHO.

[jrfnl]

@tomsommer Anyone can work on this, it's open source, so contributions are welcome.

[tomsommer]

For info, WP 6.4 issues mentioned in the forums in regards to timeouts, REST API problems in Site Health and issues reaching WP plugin/update servers are probably related to this issue (since curl fails on remote endpoints).
https://wordpress.org/support/forum/how-to-and-troubleshooting/
https://wordpress.org/support/forum/installation/

Even if you fix this now the issue prevents any future auto-upgrade to a 6.4.1, since it breaks Curl requests, so the only way for people to update would be manually. The longer you wait, the bigger the problem will become. Just FYI.

We are applying a patch to our 6.4 customers which reverts #657 directly in the 6.4 source, this solves the issue. I recommend others do this as well.

[La-Geek]

We are applying a patch to our 6.4 customers which reverts #657 directly in the 6.4 source, this solves the issue.

And how will the related websites receive this update? 6.4 is already installed and Updates are not possible?

[laszlof]

Chiming in here. We're also seeing this widespread across our platform with sites on 6.4. Using the patch @tomsommer suggested seems to resolve the issue.

If this indeed only affects curl 7.29, it seems the proper fix would be to add another condition to limit the scope of that header change.

However, I'm concerned how this is even going to get applied to sites now without manual intervention.

[laszlof]

@jrfnl I'm following the PR trail for this change. It looks like the original submitter was having issues with Wordpress updates on a very specific environment (with an IPS installed). I'm curious how this was ever tested, as none were ever provided, and very little information (if any) is available on what exactly was broken with curl to begin with.

I feel like the original PR here should be a straight rollback. There is a very real issue present now that is extremely impactful and easily reproducible, which cannot be said for the original issue that was raised.

I've opened #839 to address this.

[adrian-enspired]

extremely impactful

tens of thousands of sites for us. i'm sure many more elsewhere.

[portavoznet]

Can't update sites. Autoupdating WordPress core is marked now as default... Please, revert it... Problems after updating to WordPress 6.3 and more problems after updating to WordPress 6.4.

[gregoriopellegrino]

Same error here. You can temporary fix it changing line 367 in wp-include/Requests/src/Transport/Curl.php from:

if ($this->version < self::CURL_7_22_0 && !isset($headers['Connection'])) {

to:

if (!isset($headers['Connection'])) {

[ajtruckle]

Can't update sites. Autoupdating WordPress core is marked now as default... Please, revert it... Problems after updating to WordPress 6.3 and more problems after updating to WordPress 6.4.

Ditto.

[javiermartingonzalez]

Same here! All my websites locked after updating to Wordpress 6.4. Those one without update, working as normally.

When trying to ejecute some Ajax calls (backups, WP updates)... it gives a timeout. For WP updates the .maintenance file need to be removed manually.

I'm using CentOS7 with Plesk 18.0.56 .

@gregoriopellegrino change solves the problem

[laszlof]

Heres the related bug report in trac: https://core.trac.wordpress.org/ticket/59842

[schlessera]

Given the impact, this should be hotfixed right away in WP Core (not waiting for a Requests release) and then we need to talk to the WP systems team to see how we can get that hotfix onto the running systems. I assume that this might be something that can be fixed at the server level for api.wordpress.org.

[La-Geek]

I changed Curl.php (path /wp-includes/Requests/src/Transport/Curl.php)

as explained there. I am attaching the modified file as a zip to rule out that I have made a mistake.

Edit:
Patched (correct) file:
Curl.zip

Edit:
This fixed the issue. The error in Site-Health and the update problem disappear.

My Curl version: 7.29.0 NSS/3.53.1

[tiffanybridge]

It would also cause problems with payment plugins trying to reach accounts: woocommerce/woocommerce-gateway-stripe#2751

[RiodeJaneiroo]

i have the same problem. I'm back on 6.3.

[bbus99172]

Is there an ETA for this to be done? My 4 domains all suffer from this error 28, and I've been upgrading/installing support files, and it sounds like something is wrong with the wp scripts. What should one do to fix this? Just dont want to much up code somewhere, so assistance is appreciated :)

Brian

[laszlof]

Is there an ETA for this to be done? My 4 domains all suffer from this error 28, and I've been upgrading/installing support files, and it sounds like something is wrong with the wp scripts. What should one do to fix this? Just dont want to much up code somewhere, so assistance is appreciated :)

Brian

You'll want to edit wp-includes/Requests/src/Transport/Curl.php in your wordpress installations with the changes outlined in #839

[bbus99172]

Thanks for the help but if I try to edit anything, I want to know WHAT to edit, and WHERE to edit it/change it. I dont DARE fool with the code, cause I am not sure what to change, I am sorry about that, but I need guidance, since it looks like gibberish, and I have curl 7.74.0 {debian u10) and I need some help reading the changes to know what the error is, and how to edit it right so it will work OK :)

Brian

[bbus99172]

If I click the #839 all that does is gives me a PULL request, and NOT a file fix, and I look at the file with all the colors and i cant tell what to change......new to editing code like this: I think editing wp-config.php is easier than this :(

Brian

[tomsommer]

Guys please keep the Issue clean of support stuff, use https://wordpress.org/support/forums/ for this. Thanks.

[bbus99172]

So, how do I make this work: this is as confusing; do I just replace the bad file with curl.zip? Plus: I have

curl 7.74.0 (x86_64-pc-linux-gnu) libcurl/7.74.0 OpenSSL/1.1.1w zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2020-12-09

and not 7.2, and that file looks complicated

Brian

[jbcaprell]

Guys please keep the Issue clean of support stuff, use https://wordpress.org/support/forums/ for this. Thanks.

Would it be helpful to put the patch directions in your main post? You and I know how GitHub works, but there’s going to be a lot of traffic to this issue because it’s such a critical breakage, and they’re not going to care that this isn’t a support thread. Something to the effect of, before the issue body:

If you are just looking to fix the cURL timeouts issue in WordPress 6.4, replace:

/wp-includes/Requests/src/Transport/Curl.php

… with the following file:

Curl.php.zip

[desrosj]

There are currently no patch suggestions. The code you're seeing will fix the issue, but it's not something that would be recommended for everyone. The team is working to figure out a way to fix this, ideally without the need for anyone to do anything. No recommendations will be made until the factors are understood and there's a plan to move forward.

[tomsommer]

@jbcaprell will do

[itsensellc]

Thanks for the help but if I try to edit anything, I want to know WHAT to edit, and WHERE to edit it/change it. I dont DARE fool with the code, cause I am not sure what to change, I am sorry about that, but I need guidance, since it looks like gibberish, and I have curl 7.74.0 {debian u10) and I need some help reading the changes to know what the error is, and how to edit it right so it will work OK :)

Brian

This states it well: #838 (comment). You have no choice but to edit the file, or revert back to your backup.

[schlessera]

This was fixed in a hotfix release for Requests was published: https://github.com/WordPress/Requests/releases/tag/v2.0.9 by reverting PR #657 .

All further discussion about the implications for WP Core 6.4 and its inability to be updated should be discussed on the WordPress project. The Trac ticket can be found at https://core.trac.wordpress.org/ticket/59842 and further discussion happens in both #core and #core-upgrade-install channels.

[bbus99172]

Still have the error 28: PLEASE FIX this :( I went and put the NEW Curl.php.zip at /wp-includes/Requests/src/Transport/Curl.php

and I STILL have:

The REST API is one way that WordPress and other applications communicate with the server. For example, the block editor screen relies on the REST API to display and save your posts and pages.

When testing the REST API, an error was encountered:

REST API Endpoint: https://buddy-baker.info/index.php/wp-json/wp/v2/types/post?context=edit
REST API Response: (http_request_failed) cURL error 28: Resolving timed out after 10000 milliseconds

Somebody FIX this: websites wont even function!!

Brian

[jbcaprell]

Still have the error 28: PLEASE FIX this :( I went and put the NEW Curl.php.zip at /wp-includes/Requests/src/Transport/Curl.php

and I STILL have:

The REST API is one way that WordPress and other applications communicate with the server. For example, the block editor screen relies on the REST API to display and save your posts and pages.

When testing the REST API, an error was encountered:

REST API Endpoint: https://buddy-baker.info/index.php/wp-json/wp/v2/types/post?context=edit REST API Response: (http_request_failed) cURL error 28: Resolving timed out after 10000 milliseconds

Somebody FIX this: websites wont even function!!

Brian

This isn’t the place for support requests; that said, for clarity, you replaced the existing Curl.php file with the one inside the .zip archive, right?

Knock on wood, the WordPress core team will figure something out soon such that this isn’t necessary; if you’re not comfortable making this fix, you should really wait.

[bbus99172]

Yes: I copied Curl.php.zip to

/var/www/mallard.dkpi/public/wp-includes/Requests/src/Transport$

The old file is RENAMED:

-rw-r--r-- 1 root root 19571 Nov 6 12:27 Curl-BROKEN.php
-rw-r--r-- 1 root root 20575 Nov 8 13:44 Curl.php

if the file is where its supposed to be, why do I STILL get REST API and download crashes???

Come on now, The file did nothing for me, and ive restarted server 3 times

Brian

[bbus99172]

NOT Fixed :( Done everything I was asked to do, STILL hanging and erroring ;(

[schlessera]

The WordPress Core 6.4.1 release including this fix is scheduled to be released tomorrow Nov 9 @ 00:00 UTC.

The RC1 is already available for testing: https://make.wordpress.org/core/2023/11/08/wordpress-6-4-1-rc1-is-now-available/

[jordantrizz]

@jrfnl I agree with that, I suggest the author of #657 should account for that (or the person who merged this)? Right now the change breaks previously working versions of Curl, versions which are bundled with production PHP versions (Remi PHP) and Operating Systems (RHEL7/CentOS 7), which makes this bug more important - IMHO.

My comment isn't helpful, but the following should be considered.

curl version 7.29.0 was released on February 6 2013. The following 78 security problems are known to exist in this version.

https://curl.se/docs/vuln-7.29.0.html

I understand some providers are still running CentOS7 and RHEL7 still in production, but why?

There's a goal to move to PHP 8.1 where most Linux distributions are running curl 7.6+

[itsensellc]

@jrfnl I agree with that, I suggest the author of #657 should account for that (or the person who merged this)? Right now the change breaks previously working versions of Curl, versions which are bundled with production PHP versions (Remi PHP) and Operating Systems (RHEL7/CentOS 7), which makes this bug more important - IMHO.

My comment isn't helpful, but the following should be considered.

curl version 7.29.0 was released on February 6 2013. The following 78 security problems are known to exist in this version.

https://curl.se/docs/vuln-7.29.0.html

I understand some providers are still running CentOS7 and RHEL7 still in production, but why?

There's a goal to move to PHP 8.1 where most Linux distributions are running curl 7.6+

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

[jordantrizz]

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.

We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

[laszlof]

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.

We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

I'm going to chime in here, as I work for one of the hosting providers affected by this.

1. The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.
2. This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.
3. No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.
4. Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.
5. Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.
6. I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.

Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

[calvinalkan]

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.
We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

I'm going to chime in here, as I work for one of the hosting providers affected by this.

1. The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.

2. This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.

3. No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.

4. Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.

5. Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.

6. I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.

Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

No patch has been back ported since at least two years.

https://git.centos.org/rpms/curl/releases

[laszlof]

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.
We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

I'm going to chime in here, as I work for one of the hosting providers affected by this.

1. The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.

2. This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.

3. No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.

4. Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.

5. Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.

6. I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.

Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

No patch has been back ported since at least two years.

https://git.centos.org/rpms/curl/releases

Yes, and that includes a large number of the vulnerabilities listed above.

[calvinalkan]

Yes, and that includes a large number of the vulnerabilities listed above.

Did some quick JSON filtering on: https://curl.se/docs/vuln-7.29.0.json

And it looks like 16 vulnerabilities have been released after 2022-04-27, which falls outside the last patch window.

So, yeah...

[tomsommer]

Guys. Please. This not the place to be smart on other peoples setups. Thanks.

[laszlof]

@calvinalkan Tom is right, this isn't the place. If you'd like to have a more direct conversation about this, we can certainly take it offline. You can find me on the Wordpress Make Slack as nexflaszlo

[jordantrizz]

I'm going to chime in here, as I work for one of the hosting providers affected by this.

1. The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.

CentOS 8 has cURL 7.81 available via Stream? https://centos.pkgs.org/8-stream/centos-baseos-x86_64/curl-7.61.1-18.el8.x86_64.rpm.html but updates stopped in December 2021

2. This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.

It's not the core issue, PHP versions do play a small role here. How are you providing customers PHP 8.1 on CentOS 7?

Were you not able to upgrade to CentOS 8? I know Stream was killed off in 2021, but it was released in 2019.

Siteground has PHP 7.4 with Curl 7.85.0 right now, which I'm super surprised about.

3. No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.

Totally agree.

4. Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.

See Calvin's comments.

5. Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.

I'm afraid I have to disagree; it's a mentality not a management issue. But I'm okay with disagreement; it's healthy rapport.

Who said ignore? There's a tatical approach to this, move the customers you can, deal with the 20% separately with a stated time limit. Learn why those 20% had issues migrating/updating and change up your processes and tools. Next time you attack the same problem that 20% might be 10% or 5%.

I will repeat their name since it's a lived experience, but Siteground migrated quite a few times, and it was decently efficient and error-free. But I'm sure there was some issues, it's inevitable.

Pick your poison, lost 20% or not have this issue for your entire customer base?

6. I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.

I agree too, it's a long process. But the lifecycle of upgrading and migration and has been around since the 90's. It sucks to deal with, but you have to do it; putting it off makes it worse.

I'm afraid I have to disagree with continuing to support EOL software at this scale, it's bad hygene. It's the same as WordPress supporting PHP 5.6 for years, slowing down the process of supporting PHP 8.1. If a customer chooses to run EOL software, don't punish the rest; provide options. Let the EOL customers live in their nightmare.

Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

I don't want to discredit your own experiences or your opinion. There will be tough things that we all need to do; at the end of the day, they need to get done. But putting them off to save the few doesn't make sense to me.

I think overall there's lots to learn here for everyone.

[laszlof]

@calvinalkan Theres a lot to unpack there. Hit me up on Slack and I'd be happy to discuss. I dont want to drag this issue out and muddy the waters so to speak.

I will answer one question here, as it doesn't really require much thought/discussion. We can support PHP up to 8.3 on EL7 via the remi packages that are made available. https://rpms.remirepo.net/enterprise/7/

[schlessera]

I'm going to lock the conversation in here at this point, as it went off track and doesn't relate to Requests development. The above discussions should be held in other settings.

In case something new comes up related to the above topic, please open a new issue.