Forbidden Runner version v2.317.0 is deprecated and cannot receive messages

[jonapich]

This happened to us on 2.317.0, which is the latest version at time of writing:

An error occured: Error: Forbidden Runner version v2.317.0 is deprecated and cannot receive messages.

It happened twice. Timestamps in UTC.

past occurence: #3366 (comment)

[luketomlinson]

hi @jonapich Do you have your runner's session id? Also, it seems like those logs you posted are from yesterday?

[jonapich]

@luketomlinson it did happen yesterday; sorry for the mix up. Good catch 😅

I think 8baf0469-317a-49fa-bc3e-7b835570c4f5 would be the session id.

[jonapich]

Full log line:

GET request to https://broker.actions.githubusercontent.com/message?sessionId=8baf0469-317a-49fa-bc3e-7b835570c4f5&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden

[luketomlinson]

Thanks @jonapich. This was rolled back yesterday. I'm going to close this one out.

[dtakis]

@luketomlinson can we have more details about this? We faced the same issue with v2.316.1 and upgraded to v2.317.0 to resolve this problem. What was rolled back?

[luketomlinson]

👋 @dtakis

Currently, if you register your runner with the disableupdate flag, if your runner is on an a version that is not the latest, we force you to upgrade to the latest if that version is > 30 days old. ChangeLog

The bug that was rolled back was incorrectly performing that age check on the latest version.

If you register a runner without the disableupdate flag, you should never run into this.

[jacobwgillespie]

@luketomlinson We are currently seeing this error again, here's an example session:

[2024-07-26 17:05:02Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=cfa9c468-c485-4c87-9c52-ae63a51668a6&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden

Since v2.318.0 was just released one hour ago, perhaps this has broken the logic?

[luketomlinson]

Taking a look @jacobwgillespie

[jake-setford]

@luketomlinson We are currently seeing this error again, here's an example session:

[2024-07-26 17:05:02Z ERR  GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=cfa9c468-c485-4c87-9c52-ae63a51668a6&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden

Since v2.318.0 was just released one hour ago, perhaps this has broken the logic?

Getting this aswell on all our runners using 2.317.0.

[andyspiers]

Also getting this on many repos, but not all.

We have alarms firing and production deployments that are not happening.

We have to run with updates disabled (because we run ephemeral runners in Kubernetes which start, run one job then exit) but we check for newer runner software every day and rebuild our images if a new base image is found. However given that the only newer version is 2 hours old, there is no newer container base image for us to build from.

Surely blocking the N-1 release when N is only 2 hours old is an error on Github's part?

[luketomlinson]

👋 Apologies for this error on our part. It should be rolled back now. Restarting runners should get it back up and rolling again.

[jake-setford]

@luketomlinson can confirm all runners up again thanks for sorting.

[jacobwgillespie]

Can confirm ours are working as well, thank you for the fast turnaround ❤️

[malwinders-evertz]

@luketomlinson This is happening again with the latest v2.319.1. Did try restaring the runner few times but it stops with-in seconds and throws this message.
Sep 03 10:07:48 streampro-ghrunner runsvc.sh[7881]: An error occured: Error: Forbidden Runner version v2.319.1 is deprecated and cannot receive messages.

[luketomlinson]

Hi @malwinders-evertz, do you have any runner logs, specifically the /message url that the runner is polling against?

[malwinders-evertz]

Yes,
[2024-09-03 22:24:35Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=5f0250c2-e019-433e-ac52-d61720ceacef&status=Online&runnerVersion=2.319.1&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: Forbidden

[2024-09-03 22:24:35Z ERR  Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Error: Forbidden Runner version v2.319.1 is deprecated and cannot receive messages.

[luketomlinson]

@malwinders-evertz Do you happen anything in your infrastructure (proxies/firewalls or what not) that might be throwing a 403 in between the runner and GitHub? I don't see that request make it to our servers, but I'm still digging. It's possible a firewall might be rejecting broker.actions.githubusercontent.com with a 403, which is incorrectly getting interpreted as an old runner version by the Runner itself.

[malwinders-evertz]

@luketomlinson yes, we do have a VPN setup in our infrastructure, This exact same issue occured earlier as reported, around August 13. Later on August 14 there was a github outage for almost all the services and when that outage got resolved our runner started working as expected.

[luketomlinson]

Thanks @malwinders-evertz, can you confirm that broker.actions.githubusercontent.com is allowed within your VPN? I'm not seeing this as widespread right now, and I'm suspicious that I don't see the request make it to us.

[malwinders-evertz]

Yes, i will verify that, first thing in the morning as i have to reach out to our IT team regarding this and right now no one from IT is availabe. Will get back to you in the morning.

[malwinders-evertz]

@luketomlinson I confirmed with IT, the above address was not allowed over our VPN, once they allowed it and we restarted the runner service things seems working as expected. Thanks for the quick Response. Still wondering why it was throwing Error: Forbidden Runner version v2.319.1 is deprecated and cannot receive messages

[luketomlinson]

Thanks @malwinders-evertz. I think that's an error reporting bug on the runner side. It's interpreting any 403 as a deprecated version error, which is not correct. I'm looking into fixing that.

[omgitsbillryan]

Count me in as a datapoint. Running 2.317.0 and started seeing this error at 9:00am this morning.

[pavanpatilingenovis]

Had the same issue here.

Sep 18 15:34:29 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: Started running service
Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: √ Connected to GitHub
Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: Current runner version: '2.317.0'
Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: 2024-09-18 15:34:31Z: Listening for Jobs
Sep 18 15:34:31 vmss-xxx-xx-xx-eastus000001 runsvc.sh[358641]: An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.

I updated '2.319.0' ...... still same issue

[devAl1]

I'm using Helm's latest version 0.9.3 and get this issue. Here are some logs, sigh.

√ Connected to GitHub
[RUNNER 2024-09-19 16:52:42Z INFO Terminal] WRITE LINE:

[RUNNER 2024-09-19 16:52:42Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-09-19 16:52:42Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2024-09-19 16:52:43Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2024-09-19 16:52:43Z INFO MessageListener] Session created.
[RUNNER 2024-09-19 16:52:43Z INFO Terminal] WRITE LINE: Current runner version: '2.319.0'
Current runner version: '2.319.0'
[RUNNER 2024-09-19 16:52:43Z INFO Terminal] WRITE LINE: 2024-09-19 16:52:43Z: Listening for Jobs
2024-09-19 16:52:43Z: Listening for Jobs
[RUNNER 2024-09-19 16:52:43Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-09-19 16:52:43Z ERR  GitHubActionsService] GET request to https://pipelinesghubeus10.actions.githubusercontent.com/ziWsyY3jNadAyEohLfZPgIwCZR6CEEED8DQ8cYkMpF1XbwcOFv/_apis/distributedtask/pools/1/messages?sessionId=418d28f7-7aaf-43a9-862c-699130c5d94e&status=Online&runnerVersion=2.319.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-09-19 16:52:43Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-09-19 16:52:43Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-09-19 16:52:43Z ERR  Terminal] WRITE ERROR: An error occured: Runner version v2.319.0 is deprecated and cannot receive messages.

[luketomlinson]

👋 From our docs, for a given version, once a newer version is made available, there are 30 days until that version is deprecated.

v2.319.1 is the latest version. Anything older than that is now outside the supported window.

[makashu]

Snippet from the Support Policy notes the following:

We support (at maximum) 2 GA images and 1 beta image at a time. We begin the deprecation process of the oldest image label once the newest OS image label has been released to GA

So the documentation here https://docs.github.com/en/enterprise-cloud@latest/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#controlling-runner-software-updates-on-self-hosted-runners seems conflicting this doc https://github.com/actions/runner-images/?tab=readme-ov-file#software-and-image-support?

Does these docs need to be in-sync?

[devAl1]

Got it. Turns out we were using a custom runner based upon the runner image. So we just regenerated the runner image and we were good. I initially missed the minor version being off.

[V-know]

We encountered the same issue sine 2 days ago, the error log is:

ERR GitHubActionsService] GET request to https://pipelinesghubeus22.actions.githubusercontent.com/pLDjbMqlMDBG5S74r6lIYrddOvCnPtmrY70Us9frMAg06mRSm0/_apis/distributedtask/pools/15/messages?sessionId=4a970bbe-5fa5-458e-905e-c284c29dd85e&status=Online&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true failed. HTTP Status: Forbidden
[RUNNER 2024-09-25 10:43:26Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
[RUNNER 2024-09-25 10:43:26Z INFO Runner] Deleting Runner Session...
[RUNNER 2024-09-25 10:43:26Z ERR Terminal] WRITE ERROR: An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
An error occured: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-25 10:43:26Z ERR Listener] GitHub.DistributedTask.WebApi.AccessDeniedException: Runner version v2.317.0 is deprecated and cannot receive messages.
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-09-25 10:43:26Z ERR Listener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)

Just now we upgaded the runner version from v2.317.0 to v2.319.1, all errors disappeared.
Our runners started working as expected.