SyncRepositoryHooks failing due to EINTR

[sshaikh]

• Gitea version (or commit ref): 1.12.0
• Git version: 2.20.1
• Operating system: Debian
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

2020/05/25 18:52:08 ...dules/setting/git.go:93:newGit() [I] Git Version: 2.20.1, Wire Protocol Version 2 Enabled
2020/05/25 18:52:08 .../repository/hooks.go:156:SyncRepositoryHooks() [T] Doing: SyncRepositoryHooks
2020/05/25 18:52:08 ...m.io/xorm/core/db.go:154:QueryContext() [I] [SQL] SELECT "id", "owner_id", "owner_name", "lower_name", "name", "description", "website", "original_service_type", "original_url", "default_branch", "num_watches", "num_stars", "num_forks", "num_issues", "num_closed_issues", "num_pulls", "num_closed_pulls", "num_milestones", "num_closed_milestones", "is_private", "is_empty", "is_archived", "is_mirror", "status", "is_fork", "fork_id", "is_template", "template_id", "size", "is_fsck_enabled", "close_issues_via_commit_in_any_branch", "topics", "avatar", "created_unix", "updated_unix" FROM "repository" WHERE id>$1 LIMIT 50 [0] - 7.145921ms
2020/05/25 18:52:08 main.go:111:main() [F] Failed to run app with [./gitea admin --config /home/gitea/gitea-config/gitea/app.ini regenerate hooks]: SyncRepositoryHook: write old hook file '/appdata/repositories/config.git/hooks/pre-receive': close /appdata/repositories/config.git/hooks/pre-receive: interrupted system call

Description

After moving my app.ini, I needed to sync hooks. This failed from webgui so did it from command line:

./gitea admin --config /home/gitea/gitea-config/gitea/app.ini regenerate hooks

Thais also fails with the above error, although it appears that some hooks may have changed before the failure. Thus repeated runs of the above will eventually get through all the hooks (as the order of attempts are not fixed).

Environment

I'm running Gitea in an LXC container, and my repos are on a bind mount (which in turn is an autofs CIFS mount on the LXC host). The repo files are mounted -rwxr-xr-x and owned by the same user that runs gitea.

[zeripath]

So this has to do with changes in go 1.14 that mean that CIFS has problems.

See:
golang/go#39026 (comment)
golang/go#38033
golang/go#20400

It appears that GODEBUG=asyncpreemptoff=1 will reduce this from happening.

Rather disappointingly:

Sorry, we aren't going to backport the EINTR change to 1.14. That is much too invasive and risky.

1.14 did not introduce this problem, it merely made it occur more often. The workaround of setting GODEBUG will reduce the number of times the problems occurs back to 1.13 levels.

It looks like the only way to fix this to either:

• You need to set GODEBUG=asyncpreemptoff=1 in the environment (!)(!)
• Wait till go 1.15 (!)(!)
• Rewrite every call that could return an EINTR to retry (!)(!)

this is highly irritating.

[sshaikh]

Ill have a play - this may be why I'm seeing another issue:

#11606

[zeripath]

I suspect it's the same underlying issue.

[sshaikh]

I can confirm that setting GODEBUG=asyncpreemptoff=1 fixed at least this issue.

Will monitor the other behaviours I noticed going forward.

[zeripath]

I'm sorry it's a crap "solution" hopefully go 1.15 will fix it properly otherwise we're gonna have to wrap every call that could return an EINTR which will be beyond tiresome!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[zeripath]

It would be useful to know if this is still happening as we've done a number of things to avoid this.

[sshaikh]

So, update to latest version and remove GODEBUG=asyncpreemptoff=1?

If so I can give it a go over the next few days.

[zeripath]

yeah that would be helpful

[sshaikh]

I removed the GODEBUG and reproduced the issue with 1.12.

I then updated to 1.12.5 and did a repo sync, but unfortunately got the same "interrupted system call" error. I'll keep it as is for the time being to see if I get those random 500s too.

[zeripath]

are you building with Go 1.15.x

[sshaikh]

I used the binaries from the downloads page.

[sshaikh]

Oh and I have seen the random 500s since too, so I've readded the env variable (which has done the tick again).

[sshaikh]

I've removed GODEBUG=asyncpreemptoff=1 with 1.13.1 and can't see any immediate signs of the interruptions. I'll monitor and update if I see any further issues.

[zeripath]

OK I'll remove the confirmed label so this can go stale and we can close it if so

[sshaikh]

Confirmed I'm afraid:

2021/01/09 18:13:31 routers/repo/repo.go:183:handleCreateError() [E] CreatePost: initRepository: createDelegateHooks: write new hook file '/appdata/repositories/user/repo.git/hooks/update.d/gitea': close /appdata/repositories/user/repo.git/hooks/update.d/gitea: interrupted system call

[zeripath]

if you confirm you have to tell us what version you are running.

[sshaikh]

Version unchanged from my last report at 1.13.1.

[zeripath]

damn.

[lunny]

Have you stopped Gitea when you execute that command line?

[sshaikh]

No, I haven't. But:

1. The original issue occurred when using the GUI for a variety of typical operations (like creating a new repo or syncing hooks).
2. The GODEBUG solution solves the issue everywhere.

I don't actually use the command line, but so far it's been 100% correlated with the real GUI issues so is a deterministic enough test.

When I reconfirmed the issue on the 9th, it was after experiencing the issue creating a repo via the web, after which I recreated the issue on the command line in order to provide logs here.