Error migrating Kong OSS from 2.8.3 to 3.1.1.

[mayankmurari]

Discussed in #11421

^{Originally posted by mayankmurari August 17, 2023}
I am trying to migrate Kong from 2.8.3 to 3.1.1. Kong 3.1.1 pods are not coming up and below error is present in the logs.
Current Kong version is 2.8.3 with KIC 2.8.2 and Postgres DB.

 [emerg] 1#0: bind() to unix:/kong_prefix/worker_events.sock failed (98: Address in use)
 [notice] 1#0: try again to bind() after 500ms```</div>

[bungle]

@mayankmurari, is it possible for you to test 3.4.0, just to check if the issue has already been fixed? I remember we have done some changes there.

[mayankmurari]

Hi @bungle
I was able to upgrade Kong from 2.8.3 to 3.0.1 without any issues and I am running with Kong 3.0.1 now.

After that I tried upgrading from 3.0.1 to 3.1.1 and got the same error.

2023/08/28 12:29:50 [emerg] 1#0: bind() to unix:/kong_prefix/worker_events.sock failed (98: Address in use)
nginx: [emerg] bind() to unix:/kong_prefix/worker_events.sock failed (98: Address in use)
2023/08/28 12:29:50 [notice] 1#0: try again to bind() after 500ms

Then as per you recommendation I tried upgrading to 3.4.0 from 3.0.1 and it worked. So now as per Kong documentation below is the recommended upgrade path and if we jump from 2.8.3-->3.0.1-->3.4.0 is it still recommended?

2.8.x Traditional No Upgrade to 3.1.1.3, upgrade to 3.2.x, upgrade to 3.3.x, and then upgrade to 3.4.x.
Kong Documentation

[jeremyjpj0916]

Would also be interested to know if docs vs whats actually feasible are not in order w minimal stepping stones. maybe 2.8.x latest to 3.4.0 possible?

[hanshuebner]

@outsinre Do you have an answer for @jeremyjpj0916 ?

[outsinre]

@hanshuebner from my memory, this is a bug. Probably upgrading to newer versions can help.

Will locate the fix commit.

[hanshuebner]

@outsinre So you mean upgrade from 2.8.3 to 2.8.4 and then to 3.4.1?

[outsinre]

@hanshuebner seems not an upgrade issue.

Seems already fixed in 3.0: #9254. The root cause is the previous running socket file is not removed.

This is weird. If it is fixed in 3.0, why not 3.1.1.

---

After that I tried upgrading from 3.0.1 to 3.1.1 and got the same error.

Probably the kong prefix is mounted or a volume?

A temporary solution is to manually remove the existed socket file. Then start kong.

[Tieske]

seems related to internal ticket: https://konghq.atlassian.net/browse/KAG-328

[mayankmurari]

@outsinre
I do see issue with 3.0.1 as well sometimes and after restarting kong pod it goes away.
Tried same with 3.2.2 and I see dangling sockets are getting cleaned up and "address in use" error is not coming.