Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

程序意外退出锁没有释放的问题 #50

Open
heyz opened this issue Nov 24, 2020 · 3 comments
Open

程序意外退出锁没有释放的问题 #50

heyz opened this issue Nov 24, 2020 · 3 comments

Comments

@heyz
Copy link

heyz commented Nov 24, 2020

  某个worker调度任务时通过DabaseLock.TryGetLock()将schedulelocks字段status设置为1表示获取到锁,

如果在没有执行DabaseLock.Dispose()方法之前worker崩溃意外退出;
这样会导致schedulelocks中的status没有设置为0,也就是锁没有被释放,当再次启动worker会发现任务永远不会被调度,因为DabaseLock.TryGetLock获取不到锁。
目前我临时的解决方法是在管理后台点击【停止】按钮时将该任务的锁释放掉。博主有好的解决方案没?

@hey-hoho
Copy link
Owner

目前这种基于数据库的分布式锁实现确实有这个不能释放的问题,简单的处理办法可以加一个后台扫描线程,把持有锁超过X时间的强制释放掉。
因为后期规划了使用redis来实现这一块,所以暂时没有优化。
有兴趣可以参与进来。

@guangmingwan
Copy link

mysql的话可以加事务避免出现僵尸锁,测试过,拔网线和杀死worker都能避免,talk is cheap ,show my code:
RootJob.cs的Execute改造:

public async Task Execute(IJobExecutionContext context)
{
    IJobDetail job = context.JobDetail;
    if (job.JobDataMap["instance"] is IHosSchedule instance)
    {
        _sid = Guid.Parse(context.JobDetail.Key.Name);
        
        using (var scope = new ScopeDbContext())
        {
            _tracer = scope.GetService<RunTracer>();
            var locker = scope.GetService<HosLock.IHosLock>();


            // 预防出现僵死锁,开始事务(使用DbContext的异步事务处理)
            using (var transaction = await scope.GetDbContext().Database.BeginTransactionAsync())
            {
                try
                {
                    //设置锁超时1秒
                    scope.GetDbContext().Database.ExecuteSqlRaw(@"SET SESSION innodb_lock_wait_timeout = 1");
                    if (locker.TryGetLock(context.JobDetail.Key.Name))
                    {
                        await InnerRun(context);
                        // 如果InnerRun成功,则提交事务
                        await transaction.CommitAsync();
                    }
                    else
                    {
                        //await transaction.RollbackAsync();
                        throw new JobExecutionException("lock_failed");
                    }
                }
                catch (Exception e)
                {
                    
                    LogHelper.Error($"任务\"{instance.Main.Title}\"-{job.Key}拿锁失败!", e);
                    // 出现异常时,事务应自动回滚,但显式调用RollbackAsync以确保事务状态明确
                    if (e is JobExecutionException)
                    {
                        //这个错误不会滚
                    }
                    else
                    {
                        try
                        {
                            await transaction.RollbackAsync();
                        }
                        catch (Exception rollbackEx)
                        {
                            LogHelper.Error("事务回滚时发生错误!", rollbackEx);
                        }
                    }

                    throw e;
                }
                finally
                {
                    // 恢复 lock wait timeout 为默认50(mysql 8.0默认为50秒)
                    scope.GetDbContext().Database.ExecuteSqlRaw(@"SET SESSION innodb_lock_wait_timeout = 50");
                }
            }
        }
    }
}

@huangyong2016
Copy link

huangyong2016 commented Jun 6, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants