1.18.0 on Windows error: can not resolve localhost, ! [remote rejected] master -> master (pre-receive hook declined)

[jinyangcruise]

Description

Save your time:

1.17.4 is good but 1.18.0 has a bug: 
dial tcp: lookup localhost on 114.114.114.114:53: no such host

Detailed:

I use 1.18.0 Gitea, when I push files, occur:

$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 20 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 305 bytes | 305.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Gitea: Internal Server Error
remote: Unable to contact gitea: Post "http://localhost:3000/api/internal/hook/pre-receive/jinyang/test": dial tcp: lookup localhost on 114.114.114.114:53: no such host
To http://localhost:3000/jinyang/test.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'http://localhost:3000/jinyang/test.git'

I searched some related issues but finally failed to fix this.

Then I installed 1.17.4 Gitea and this problem gone.

Then I re-installed 1.18.0 Gitea again and still can not push.

Gitea Version

1.18.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

NA

Git Version

git version 2.39.0.windows.2

Operating System

WINDOWS 11

How are you running Gitea?

download binary on https://dl.gitea.io/gitea
[gitea-1.18.0-windows-4.0-amd64.exe]

download it on my PC and move it to D:\Program Files\Gitea, then double click the gitea-1.18.0-windows-4.0-amd64.exe

Database

SQLite

[lunny]

Could you ping localhost?

[wxiaoguang]

Workaround: set LOCAL_ROOT_URL with 127.0.0.1, do not use localhost

[jinyangcruise]

Could you ping localhost?

yes

[jinyangcruise]

Workaround: set LOCAL_ROOT_URL with 127.0.0.1, do not use localhost

I can't find a LOCAL_ROOT_URL in app.ini, should I add it at somewhere?

[lunny]

Workaround: set LOCAL_ROOT_URL with 127.0.0.1, do not use localhost

I can't find a LOCAL_ROOT_URL in app.ini, should I add it at somewhere?

Please check the documentation.

[jinyangcruise]

Workaround: set LOCAL_ROOT_URL with 127.0.0.1, do not use localhost

I can't find a LOCAL_ROOT_URL in app.ini, should I add it at somewhere?

Please check the documentation.

Thank you very much! Success!

By the way, english version documentation has this LOCAL_ROOT_URL but chinese version not.
https://docs.gitea.io/en-us/config-cheat-sheet/#server-server
https://docs.gitea.io/zh-cn/config-cheat-sheet/#server-server

[emehmet]

it does not work for me

[starpeng]

it work for me

Workaround: set LOCAL_ROOT_URL with 127.0.0.1, do not use localhost

[jinyangcruise]

it does not work for me

workaround:

LOCAL_ROOT_URL = http://127.0.0.1:3000/

[lunny]

it does not work for me

Is your problem the same as others?

[zeripath]

So leaving aside the workaround which I am sure will fix the issues people are describing - I don't understand why it is needed or what has changed to make it needed.

As far as I can see the determination of the value of setting.LocalURL (i.e. that which LOCAL_ROOT_URL maps to) has not changed and nor has the dialer for this code.

So I don't understand why this should suddenly be a problem...

[zeripath]

For others seeing this conversation the LOCAL_ROOT_URL is in the [server] section of the app.ini and it must be added there.

[emehmet]

Hey @jinyangcruise
It worked as you said.
Thank you! 👍

it does not work for me

workaround:

LOCAL_ROOT_URL = http://127.0.0.1:3000/

[wxiaoguang]

I think this issue should be re-opened and it needs a patch.

To make the workaround prominent, I didn't say too much about my guess before.

Here is my opinion of this bug:

How it happens?

Gitea (Golang net package) can not resolve host localhost.

Golang's net package need correct /etc/hosts and /etc/nsswitch.conf to resolve domains correctly. If anything is broken, it may not resolve some domains. On Windows, it also depends on C:\Windows\System32\Drivers\etc\hosts file

Why there are so many reports recently?

Possibility 1

Alpine / Go or Go for Windows changed some behaviors which conflict with user's environment, then Golang's net package doens't work well in some rare cases (no evidence yet). But since some users reported that 1.17.x works but 1.18.x fails, maybe there is something changed.

Possibility 2

Many users' systems were already broken, they messed up their hosts file, they might have deleted the 127.0.0.1 localhost line. I know a lot of such users.

Before, some public DNS servers can also resolve localhost, for example, there is still one now:

$ dig localhost @223.5.5.5
;; ANSWER SECTION:
localhost.		2495	IN	A	127.0.0.1

Recently some public DNS servers might have stopped such non-standard behavior, if they do not resolve localhost anymore, the users with broken localhost definition will encounter the problem.

So, some users with broken hosts files can get Gitea working before, but get errors now.

The patch

Approach 1

Use 127.0.0.1 instead of localhost for the default value of LOCAL_ROOT_URL

• Pros

  • 127.0.0.1 is the de-facto standard loopback IP, it should exist in every OS
  • won't be affected by Host OS/Alpine/Golang anymore (in most cases)
  • even if the user's hosts file is broken, it still works

• Cons

  • In a pure IPv6 network, the 127.0.0.1 may not exist, but the pure IPv6 network should be pretty rarer than broken hosts files, and a user with pure IPv6 must be an experienced user, they should be able to set LOCAL_ROOT_URL to a IPv6 loopback without any question.

Approach 2

Find the root case why 1.17.x works but 1.18.x doesn't.

Update: maybe it's a Go 1.19 bug: golang/go#53490

And: https://tip.golang.org/doc/go1.19#net

The net package now has initial support for the netgo build tag on Windows. When used, the package uses the Go DNS client (as used by Resolver.PreferGo) instead of asking Windows for DNS results. The upstream DNS server it discovers from Windows may not yet be correct with complex system network configurations, however.

Removing the netgo tag from Windows build may resolve the problem.

Approach 3

Try to resolve localhost at startup, report fatal errors if the resolution fails.

[zeripath]

We could use 127.0.0.1 instead of localhost only on windows then we get none of the cons.

[zeripath]

But yes I agree I think this was closed too soon and we need to add a patch - if not for the users, then just to save ourselves time.

[DuaneDu]

@zeripath
Config is

[server]
LOCAL_ROOT_URL = http://127.0.0.1:3000/
SSH_DOMAIN       = localhost
DOMAIN           = 192.168.2.50
HTTP_PORT        = 3000
DISABLE_SSH      = false
SSH_PORT         = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = D:/work/gitea/data/lfs
LFS_JWT_SECRET   = TGUsxz3aFCZEj8CnOPFzEPUXLhzRTAlOxvd8Rwvl9Q4
OFFLINE_MODE     = false

error

console log

gitea.log
gitea.log

[KazzmanK]

Same problem for me, windows install, for unknown reason problem raised today, yesterday it was ok.
in hosts we have 127.0.0.1 localhost record. It still calls for dns and fails.
workaround with LOCAL_ROOT_URL helped.

@DuaneDu , add DEV run mode, client will show bit more detailed error message

[zeripath]

@DuaneDu your logs now indicate that things are should now be working correctly for you.

Your pre-receive hook is being declined by the user does not have permission to that repo

[wxiaoguang]

I think it's related to this Go bug:

• net: 1.19 DNS lookups broken on Windows if standard library built with CGO_ENABLED=0 golang/go#53490

And: https://tip.golang.org/doc/go1.19#net

The net package now has initial support for the netgo build tag on Windows. When used, the package uses the Go DNS client (as used by Resolver.PreferGo) instead of asking Windows for DNS results. The upstream DNS server it discovers from Windows may not yet be correct with complex system network configurations, however.

---

Removing the netgo tag from Windows build may resolve the problem.

[natsutteatsuiyone]

This webhook error may be related to this issue.

2023/01/11 23:20:28 ...s/webhook/webhook.go:122:handle() [E] [63bec58b-9] webhook.Deliver failed: Post "http://dev-738-11/gitea-webhook/post": dial tcp: lookup dev-738-11 on 192.168.10.3:53: server misbehaving

https://gist.github.com/natsutteatsuiyone/4c7b8e3b444a61080bb04ece20b316aa

[DuaneDu]

@zeripath

1. This account is administrator
2. this repo is public
3. this repo is created by me
4. I have all permission on this repo

RUN_MODE=dev
gitea.log
router.log

RUN_MODE=test

gitea.log
router.log

[zeripath]

The net package now has initial support for the netgo build tag on Windows. When used, the package uses the Go DNS client (as used by Resolver.PreferGo) instead of asking Windows for DNS results. The upstream DNS server it discovers from Windows may not yet be correct with complex system network configurations, however.

It certainly appears that the upstream DNS server it is discovering is doing the incorrect thing for at least localhost...

Removing the netgo tag from Windows build may resolve the problem.

I guess the question is what does the netgo tag do on Windows build and is it really necessary to make a "static" build with it for windows? Removing netgo here:

gitea/Makefile

Lines 759 to 764 in 16e9dec

	.PHONY: release-windows
	release-windows: | $(DIST_DIRS)
	CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -buildmode exe -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-linkmode external -extldflags "-static" $(LDFLAGS)' -targets 'windows/*' -out gitea-$(VERSION) .
	ifeq (,$(findstring gogit,$(TAGS)))
	CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -buildmode exe -dest $(DIST)/binaries -tags 'netgo osusergo gogit $(TAGS)' -ldflags '-linkmode external -extldflags "-static" $(LDFLAGS)' -targets 'windows/*' -out gitea-$(VERSION)-gogit .
	endif

would create "Static" builds for windows without the use of netgo.

[zeripath]

I guess an option is to ask users who are affected if they can run Gitea with the environment variable GODEBUG=netdns=cgo, e.g.:

GODEBUG=netdns=cgo gitea web

That should force the use of the win32 api.

[wxiaoguang]

I have reported the bug to Go team (it has been confirmed as NeedsFix there)

• net: Windows build with netgo doesn't repsect hosts file, can not resolve localhost golang/go#57757

I guess the question is what does the netgo tag do on Windows build and is it really necessary to make a "static" build with it for windows? Removing netgo here:

I guess the netgo in Makefile was just a Copy&Paste and it's not necessary.

And, the netgo seemed to be a no-op on Windows before. But Go 1.19 rewrite the code for it.

So, IMO it's good to remove the netgo tag for Windows build at the moment. When Go team fix the bug, re-enable the netgo tag then (if it's really needed ...)

[zeripath]

Agreed I've just looked up the source code for golang and I've come to the same conclusion. netgo only affects lookup on windows and previously was a no-op. Therefore we can and should safely remove until/unless go can make it work correctly.

Are you able to send a PR @wxiaoguang ?

[zeripath]

As an aside I suspect the problem is here:

https://github.com/golang/go/blob/d52883f443e1d564b0300acdd382af1769bf0477/src/net/conf.go#L134-L145

	if c.netGo || r.preferGo() {
		switch c.goos {
		case "windows":
			// TODO(bradfitz): implement files-based
			// lookup on Windows too? I guess /etc/hosts
			// kinda exists on Windows. But for now, only
			// do DNS.
			fallbackOrder = hostLookupDNS
		default:
			fallbackOrder = hostLookupFilesDNS
		}
	}

That code cannot check the /etc/hosts file and thus I bet the dns lookup is going out to an external server and we're getting the external name for the host.

[zeripath]

DESCRIPTION

This bug is due to a change in Go's handling of the netgo build tag for Windows see:

#22370 (comment)

We were bitten by this because we build our static precompiled binaries with netgo for all platforms. (see #22370 (comment))

#22467 and #22468 should fix this for 1.18-dev, 1.18.1 and 1.19-dev.

WORKAROUND

For 1.18.0 set the GODEBUG=netdns=cgo environment variable when running gitea.

#22370 (comment)

Builds of what will become 1.18.1 are available now on https://dl.gitea.io/gitea/1.18