Handle UncategorizedSQLException on JdbcTemplateStorageAccessor.updateRecord()

[jaam]

Hello, we are getting UncategorizedSQLExceptions when trying to update records through JdbcTemplateStorageAccessor.updateRecord() , see stacktrace below. This doesn't happen continuously, but under certain conditions:

• During the first seconds/minutes of each month. This is probably due to our project specific backups/infrastructure activities.
• Triggered by database restarts when our infrastructure team perform PostgreSQL upgrades.

My assumption is that during those database maintenance tasks transactions are marked as readonly and therefore fail. When maintenance is completed, errors don't appear anymore in our logs. This PR adds explicit handling of UncategorizedSQLException, which in my opinion doesn't change the behavior, as locks couldn't be updated anyway. With this change, we will avoid many error log entries.

Our stack is based on PostgreSQL v11 and Cloudfoundry.

Stacktrace:

o.p.u.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
at o.p.c.v.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
at o.p.c.v.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
at o.p.c.v.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
at o.p.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at o.p.jdbc.PgStatement.execute(PgStatement.java:408)
at o.p.j.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167)
at o.p.j.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135)
at c.z.h.p.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at c.z.h.p.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at o.s.j.c.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)
at o.s.j.c.JdbcTemplate.execute(JdbcTemplate.java:651)
... 37 common frames omitted
Wrapped by: o.s.j.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [UPDATE shedlock SET lock_until = ?, locked_at = ?, locked_by = ? WHERE name = ? AND lock_until <= ?]; SQL state [25006]; error code [0]; ERROR: cannot execute UPDATE in a read-only transaction; nested exception is org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
at o.s.j.c.JdbcTemplate.translateException(JdbcTemplate.java:1542)
at o.s.j.c.JdbcTemplate.execute(JdbcTemplate.java:667)
at o.s.j.c.JdbcTemplate.update(JdbcTemplate.java:960)
at o.s.j.c.JdbcTemplate.update(JdbcTemplate.java:981)
at o.s.j.c.n.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:328)
at o.s.j.c.n.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:333)
at n.j.s.p.j.JdbcTemplateStorageAccessor.lambda$execute$0(JdbcTemplateStorageAccessor.java:117)
at o.s.t.s.TransactionTemplate.execute(TransactionTemplate.java:140)
at n.j.s.p.j.JdbcTemplateStorageAccessor.execute(JdbcTemplateStorageAccessor.java:117)
at n.j.s.p.j.JdbcTemplateStorageAccessor.updateRecord(JdbcTemplateStorageAccessor.java:82)
at n.j.s.s.StorageBasedLockProvider.doLock(StorageBasedLockProvider.java:93)
at n.j.s.s.StorageBasedLockProvider.lock(StorageBasedLockProvider.java:65)
at j.i.r.GeneratedMethodAccessor71.invoke(Unknown Source)
at j.i.r.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at j.l.reflect.Method.invoke(Unknown Source)
at o.s.a.s.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at o.s.a.f.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at jdk.proxy2.$Proxy159.lock(Unknown Source)
at n.j.s.c.DefaultLockingTaskExecutor.executeWithLock(DefaultLockingTaskExecutor.java:63)
at n.j.s.s.a.MethodProxyScheduledLockAdvisor$LockingInterceptor.invoke(MethodProxyScheduledLockAdvisor.java:86)
at o.s.a.f.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at o.s.a.f.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at o.s.a.i.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at o.s.a.f.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at o.s.a.f.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at o.s.a.f.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at c.a.b.j.e.d.r.s.ResendScheduler$$EnhancerBySpringCGLIB$$86ccdbba.executePendingSchedules(<generated>)
at j.i.r.GeneratedMethodAccessor67.invoke(Unknown Source)
at j.i.r.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at j.l.reflect.Method.invoke(Unknown Source)
at o.s.s.s.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at o.s.s.s.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at o.s.s.c.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
at j.u.c.Executors$RunnableAdapter.call(Unknown Source)
at j.u.c.FutureTask.run(Unknown Source)
at j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at j.u.c.ThreadPoolExecutor.runWorker(Unknown Source)
at j.u.c.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

[jaam]

Just to confirm the suspicion, I've found additional info in the log files that shows how connections are terminated upon PostgreSQL upgrades:

exception: o.p.u.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
message: FATAL: terminating connection due to administrator command

[lukas-krecan]

Thanks

[lukas-krecan]

Released as 5.4.0

[jaam]

Great, thanks!!