Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem running SyncthingNative on x86_64 arch (Chromebook, emulator) #583

Closed
Catfriend1 opened this issue Dec 16, 2019 · 23 comments
Closed
Assignees
Labels
Milestone

Comments

@Catfriend1
Copy link
Owner

Catfriend1 commented Dec 16, 2019

@cm-lo @decadecity

Could you please try this release and tell me if it has the same problem with SyncthingNative not starting and producing a "go native crash"?

https://github.com/Catfriend1/syncthing-android/releases/download/v1.3.3.2/com.github.catfriend1.syncthingandroid_v1.3.3.2_a0850dc1.apk

from: https://github.com/Catfriend1/syncthing-android/releases/tag/v1.3.3.2

Ref.: syncthing/syncthing-android#1451
Ref.: syncthing/syncthing-android#1703

ref: syncthing/syncthing#8710

@ghost
Copy link

ghost commented Dec 17, 2019

Syncthing-fork does not work too.

  1. v1.3.3.2_a0850dc1.log: the adb logcat;
  2. v1.3.3.2_a0850dc1.png: screenshot of the app; and
  3. v1.3.3.2_a0850dc1.txt: the app log after clicking the notification.

v1.3.3.2_a0850dc1.log

v1 3 3 2_a0850dc1

v1.3.3.2_a0850dc1.txt

@decadecity
Copy link

decadecity commented Dec 17, 2019 via email

@ghost
Copy link

ghost commented Dec 17, 2019

I found another apk in the directory (com.github.catfriend1.syncthingandroid_gplay_v1.3.3.2_a0850dc1.apk). I captured the same three items as my last post in case you need.

gplay_v1.3.3.2_a0850dc1.log
gplay_v1 3 3 2_a0850dc1
gplay_v1.3.3.2_a0850dc1.txt

@Catfriend1
Copy link
Owner Author

@imsodin The .log file posted above has the full go trace with a lot of watcher and service stuff from thejerf in it. Maybe you'll find some new info in it if you could kindly have a glance again? Does the current codebase offer a way to completely disable calling the watcher stuff (e.g. through env var or config.xml)? On the user's phone Syncthing came up fine until the ready to sync folder line and then died because of an unknown reason, maybe watcher related.
If it does not impose too much work on you I'd suggest we try making a branch/commit for testing and I make an Apk for x86_64 users. Could also be a bug introduced in v1.3.x?

@imsodin
Copy link

imsodin commented Dec 18, 2019

The log shows the same as the one in the other issue: A pipe syscall used in the notify library is failing with "bad system call". No need to even look at all the rest, that's just the usual goroutines running at the time of the crash - the origin of the crash is clear, what's unclear is why this syscall doesn't work. This is definitely no new bug on Syncthing's side, this is just a problem with the particular build. You can disable file watching in the config (per folder), you can't disable it globally.

@Catfriend1
Copy link
Owner Author

Ok, thanks for your feedback, Simon! I'll try to make a test build with disabled file watcher after christmas 👍

@Catfriend1
Copy link
Owner Author

Quote from original issue linked above:
until the app automatically upgraded itself from version 1.3.1 to version 1.3.2

I see two commits on the syncthing-android (original) repo, but don't know if one has to do with it or a go code change...

  1. syncthing/syncthing-android@c7b433b
  2. syncthing/syncthing-android@03d2f67

Between the mentioned versions, I noticed NDK was bumped from r15c to r20. We did that on the fork, too. So let's go another debug session together. I need info, if Syncthing-Fork v1.3.1 works on your x86_64 phones.

@cm-lo : Could you please try and report back if the same error also comes up when running https://github.com/Catfriend1/syncthing-android/releases/download/v1.3.1.6/com.github.catfriend1.syncthingandroid_v1.3.1.6_b6564099.apk ?

Catfriend1 added a commit that referenced this issue Dec 22, 2019
@Catfriend1
Copy link
Owner Author

I've found a dirty workaround to make SyncthingNative v1.3.3 run on x86_64.

  • cp "app\src\main\jniLibs\x86\libsyncthing.so" "app\src\main\jniLibs\x86_64\libsyncthing.so"
  • Build APK.

The part about BAD SYSCALL still wonders me.

12-22 12:49:50.831  5815     0 E Go      : SIGSYS: bad system call
12-22 12:49:50.831  5815     0 E Go      : PC=0x5e0b98f4d78b m=10 sigcode=1
12-22 12:49:50.831  5815     0 E Go      :
12-22 12:49:50.831  5815     0 E Go      : goroutine 81 [running]:
12-22 12:49:50.832  5815     0 E Go      : syscall.RawSyscall(0x16, 0xc0016632c8, 0x0, 0x0, 0x0, 0x5e0b992746f2, 0x123)
12-22 12:49:50.832  5815     0 E Go      :      E:/GitHub/syncthing-android-prereq/go/src/syscall/asm_linux_amd64.s:78 +0x2b fp=0xc001663260 sp=0xc001663258 pc=0x5e0b98f4d78b
12-22 12:49:50.833  5815     0 E Go      : golang.org/x/sys/unix.pipe(0xc0016632c8, 0x0, 0x0)
12-22 12:49:50.833  5815     0 E Go      :      E:/GitHub/syncthing-android/syncthing/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:2426 +0x4c fp=0xc0016632b0 sp=0xc001663260 pc=0x5e0b992752fc
12-22 12:49:50.834  5815     0 E Go      : golang.org/x/sys/unix.Pipe(0xc0000b2690, 0x2, 0x2, 0x0, 0x0)
12-22 12:49:50.834  5815     0 E Go      :      E:/GitHub/syncthing-android/syncthing/pkg/mod/golang.org/x/[email protected]/unix/syscall_linux_amd64.go:135 +0x67 fp=0xc0016632e0 sp=0xc0016632b0 pc=0x5e0b99274097
12-22 12:49:50.834  5815     0 E Go      : github.com/syncthing/notify.(*inotify).epollinit(0xc00012e000, 0x12, 0x0)
12-22 12:49:50.835  5815     0 E Go      :      E:/GitHub/syncthing-android/syncthing/pkg/mod/github.com/syncthing/[email protected]/watcher_inotify.go:152 +0x70 fp=0xc001663320 sp=0xc0016632e0 

@imsodin: Could it be that we need to report a bug for golang/sys/unix or update the submodule? I don't know how I could test with different versions of this.

Here are the sources I've found to follow the stacktrace:
https://github.com/syncthing/notify/blob/69c7a957d3e261f9744f46b3dd4d608d8480ad90/watcher_inotify.go#L152
-- https://godoc.org/golang.org/x/sys/unix
---- https://github.com/golang/sys/blob/master/unix/asm_linux_amd64.s

@ghost
Copy link

ghost commented Dec 22, 2019

@Catfriend1 thanks for your effort.

The apk of v1.3.1.6 and v1.3.3.2 do not work and v1.3.3.3 seems work (see screenshot). I enclosed the logcats for all three apks.

Merry Christmas!

Screenshot 2019-12-22 at 22 23 24

v1316.log
v1332.log
v1333.log

@imsodin
Copy link

imsodin commented Dec 22, 2019

Quote from original issue linked above:
until the app automatically upgraded itself from version 1.3.1 to version 1.3.2

What changed between those version is exactly that I added the x86_64 build - there's nothing to be analyzed here: The origin of is the newly added x86_64 build.

I've found a dirty workaround to make SyncthingNative v1.3.3 run on x86_64.

* cp "app\src\main\jniLibs\x86\libsyncthing.so" "app\src\main\jniLibs\x86_64\libsyncthing.so"

* Build APK.

And google play accepts that? They seem to be pretty easy to fool then o.O

@imsodin: Could it be that we need to report a bug for golang/sys/unix or update the submodule? I don't know how I could test with different versions of this.

Maybe, but we'd need to do much more digging (aka creating a minimal reproducer) as our builds involve way too many (non-standard) elements (different compiler for one). The problem might just as well be with the NDK. And I personally don't have any time for that.

@Catfriend1
Copy link
Owner Author

Catfriend1 commented Dec 22, 2019

@imsodin thanks for your reply.

And google play accepts that? They seem to be pretty easy to fool then o.O

Well, they are currently reviewing it xD. Gplay said "compliant with the 64bit policy" after upload. ;-)

I've tried with ndk15 and the problem is also there. If it's the ndk , the problem must have been there forever since.

@ghost
Copy link

ghost commented Dec 22, 2019

@Catfriend1 off topic, the reason (of the app is not running) is still shown in my v1.3.3.3 screenshot above even the app is indeed running. Apparently a tiny bug.

@Catfriend1
Copy link
Owner Author

This bug still occurs on the Android 13 emulator (AVD, x86_64 arch).

But the above mentioned workaround by copying the x86 artifact to x86_64 doesn't work. The log says "error 2 executing libsyncthingnative.so - file not found" even though the file exists in the path the log mentions. Android 12+ seem to check if the .so arch is correct before executing the file.

I've added the x86_64 build to "build-syncthing.py":

 {
        'arch': 'x86_64',
        'goarch': 'amd64',
        'jni_dir': 'x86_64',
        'cc': 'x86_64-linux-android{}-clang',
    }

Syncthing now starts, and crashes short after with this in the log: https://pastebin.com/vVVN3GbR

@Catfriend1
Copy link
Owner Author

It seems to point at the inotify watcher,but I don't understand why only this 1 (out of 4) archs isn't working an produces the "bad system call".

W/SyncthingNativeCode: SIGSYS: bad system call
W/SyncthingNativeCode: PC=0x5f82f14e9f4e m=8 sigcode=1
W/SyncthingNativeCode: 
W/SyncthingNativeCode: goroutine 96 [syscall]:
W/SyncthingNativeCode: syscall.Syscall6(0xc0004224e0?, 0x3?, 0xc0003deef0?, 0x5f82f14edeb0?, 0xc00015f748?, 0xc00015f808?, 0x0?)
W/SyncthingNativeCode: 	syscall/syscall_linux.go:90 +0x36 fp=0xc0003deea0 sp=0xc0003dee18 pc=0x5f82f15a1ff6
W/SyncthingNativeCode: syscall.Syscall6(0xe8, 0x15, 0xc0003def84, 0x1, 0xffffffffffffffff, 0x0, 0x0)
W/SyncthingNativeCode: 	<autogenerated>:1 +0x45 fp=0xc0003deee8 sp=0xc0003deea0 pc=0x5f82f15a23a5
W/SyncthingNativeCode: golang.org/x/sys/unix.EpollWait(0x5f82f14edeb0?, {0xc0003def84?, 0x0?, 0x0?}, 0x5f82f19bf0e0?)
W/SyncthingNativeCode: 	golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:56 +0x58 fp=0xc0003def58 sp=0xc0003deee8 pc=0x5f82f1922798
W/SyncthingNativeCode: github.com/syncthing/notify.(*inotify).loop(0xc000200000, 0xc0001d9300?)
W/SyncthingNativeCode: 	github.com/syncthing/[email protected]/watcher_inotify.go:188 +0x5a fp=0xc0003defc0 sp=0xc0003def58 pc=0x5f82f198d23a
W/SyncthingNativeCode: github.com/syncthing/notify.(*inotify).lazyinit.func2()
W/SyncthingNativeCode: 	github.com/syncthing/[email protected]/watcher_inotify.go:128 +0x2a fp=0xc0003defe0 sp=0xc0003defc0 pc=0x5f82f198ce6a
W/SyncthingNativeCode: runtime.goexit()
W/SyncthingNativeCode: 	runtime/asm_amd64.s:1594 +0x1 fp=0xc0003defe8 sp=0xc0003defe0 pc=0x5f82f1553a81
W/SyncthingNativeCode: created by github.com/syncthing/notify.(*inotify).lazyinit
W/SyncthingNativeCode: 	github.com/syncthing/[email protected]/watcher_inotify.go:128 +0x118
W/SyncthingNativeCode: 
W/SyncthingNativeCode: goroutine 1 [chan receive]:
W/SyncthingNativeCode: runtime.gopark(0xc00039c9e0?, 0x1?, 0x1?, 0x0?, 0x1?)
W/SyncthingNativeCode: 	runtime/proc.go:363 +0xd6 fp=0xc0007de618 sp=0xc0007de5f8 pc=0x5f82f1521e96
W/SyncthingNativeCode: runtime.chanrecv(0xc0000a09c0, 0x0, 0x1)
W/SyncthingNativeCode: 	runtime/chan.go:583 +0x49b fp=0xc0007de6a8 sp=0xc0007de618 pc=0x5f82f14ee45b
W/SyncthingNativeCode: runtime.chanrecv1(0xc00039d2f0?, 0x1?)
W/SyncthingNativeCode: 	runtime/chan.go:442 +0x18 fp=0xc0007de6d0 sp=0xc0007de6a8 pc=0x5f82f14edf58
W/SyncthingNativeCode: github.com/syncthing/syncthing/lib/syncthing.(*App).Wait(...)
W/SyncthingNativeCode: 	github.com/syncthing/syncthing/lib/syncthing/syncthing.go:376
W/SyncthingNativeCode: main.syncthingMain({{{0x0}, {0x0, 0x0}, {0xc000133e00, 0x3f}, 0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
W/SyncthingNativeCode: 	github.com/syncthing/syncthing/cmd/syncthing/main.go:686 +0xd1e fp=0xc0007dee08 sp=0xc0007de6d0 pc=0x5f82f1e3899e
W/SyncthingNativeCode: main.serveOptions.Run({{{0x0}, {0x0, 0x0}, {0xc000133e00, 0x3f}, 0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
W/SyncthingNativeCode: 	github.com/syncthing/syncthing/cmd/syncthing/main.go:411 +0x898 fp=0xc0007df218 sp=0xc0007dee08 pc=0x5f82f1e36438
W/SyncthingNativeCode: main.(*serveOptions).Run(0x1?)
W/SyncthingNativeCode: 	<autogenerated>:1 +0xa5 fp=0xc0007df478 sp=0xc0007df218 pc=0x5f82f1e3e605
W/SyncthingNativeCode: runtime.call16(0xc00033a000, 0xc0001120c0, 0x0, 0x0, 0x0, 0x8, 0xc0007df9a8)
W/SyncthingNativeCode: 	runtime/asm_amd64.s:724 +0x4b fp=0xc0007df498 sp=0xc0007df478 pc=0x5f82f1551d0b
W/SyncthingNativeCode: runtime.reflectcall(0xc00011c5a0?, 0x1?, 0x1?, 0x315b20?, 0xc0?, 0x1?, 0x1?)
W/SyncthingNativeCode: 	<autogenerated>:1 +0x3c fp=0xc0007df4d8 sp=0xc0007df498 pc=0x5f82f15562dc
W/SyncthingNativeCode: reflect.Value.call({0x5f82f20196e0?, 0x5f82f265f4e0?, 0x5f82f1e41bd0?}, {0x5f82f11828c3, 0x4}, {0xc0001f9b80, 0x0, 0x5f82f1579c88?})
W/SyncthingNativeCode: 	reflect/value.go:584 +0x8c5 fp=0xc0007dfac8 sp=0xc0007df4d8 pc=0x5f82f157ed65
W/SyncthingNativeCode: reflect.Value.Call({0x5f82f20196e0?, 0x5f82f265f4e0?, 0x5f82f1f10e20?}, {0xc0001f9b80?, 0x5f82f1eb8a20?, 0x5f82f265f588?})
W/SyncthingNativeCode: 	reflect/value.go:368 +0xbc fp=0xc0007dfb40 sp=0xc0007dfac8 pc=0x5f82f157e25c
W/SyncthingNativeCode: github.com/alecthomas/kong.callMethod({0x5f82f11822d7, 0x3}, {0x5f82f20196e0?, 0x5f82f265f4e0?, 0x3?}, {0x5f82f20196e0?, 0x5f82f265f4e0?, 0x203000?}, 0xc00031b380?)
W/SyncthingNativeCode: 	github.com/alecthomas/[email protected]/callbacks.go:95 +0x549 fp=0xc0007dfcb8 sp=0xc0007dfb40 pc=0x5f82f186dc29
W/SyncthingNativeCode: github.com/alecthomas/kong.(*Context).RunNode(0xc000134180, 0xc0001500f0, {0x0, 0x0, 0x0})
W/SyncthingNativeCode: 	github.com/alecthomas/[email protected]/context.go:755 +0x5f7 fp=0xc0007dfe80 sp=0xc0007dfcb8 pc=0x5f82f1874657
W/SyncthingNativeCode: github.com/alecthomas/kong.(*Context).Run(0x5f82f1e9e5e0?, {0x0?, 0xc00012c000?, 0x5f82f1e958e0?})
W/SyncthingNativeCode: 	github.com/alecthomas/[email protected]/context.go:780 +0x14e fp=0xc0007dfec0 sp=0xc0007dfe80 pc=0x5f82f18749ee
W/SyncthingNativeCode: main.main()
W/SyncthingNativeCode: 	github.com/syncthing/syncthing/cmd/syncthing/main.go:255 +0x3d6 fp=0xc0007dff80 sp=0xc0007dfec0 pc=0x5f82f1e35976
W/SyncthingNativeCode: runtime.main()
W/SyncthingNativeCode: 	runtime/proc.go:250 +0x213 fp=0xc0007dffe0 sp=0xc0007dff80 pc=0x5f82f1521ad3
W/SyncthingNativeCode: runtime.goexit()
W/SyncthingNativeCode: 	runtime/asm_amd64.s:1594 +0x1 fp=0xc0007dffe8 sp=0xc0007dffe0 pc=0x5f82f1553a81

@Catfriend1 Catfriend1 changed the title Problem running SyncthingNative on x86_64 arch Problem running SyncthingNative on x86_64 arch (Chromebook, emulator) Dec 7, 2022
@Catfriend1 Catfriend1 reopened this Dec 7, 2022
@Catfriend1 Catfriend1 removed the build label Dec 7, 2022
@Catfriend1 Catfriend1 removed this from the v1.3.3 milestone Dec 7, 2022
@Catfriend1
Copy link
Owner Author

@imsodin I've found the problem which causes SyncthingNative to crash when compiled against android-x86_64. I can workaround the crash by editing the following code:

"...\syncthing-android\syncthing\pkg\mod\github.com\syncthing\[email protected]\watcher_inotify.go"

  • line 184

I removed all code from the "loop" function.

func (i *inotify) loop(esch chan<- []*event) {
}

image

Then, I did the libsyncthingnative.so build for x86_64:

*** Building for x86_64
Notice: Next generation GUI will not be built; see --with-next-gen-gui.
github.com/syncthing/notify
github.com/syncthing/syncthing/lib/fs
github.com/syncthing/syncthing/lib/locations
github.com/syncthing/syncthing/lib/osutil
github.com/syncthing/syncthing/cmd/syncthing/cmdutil
github.com/syncthing/syncthing/lib/ignore
github.com/syncthing/syncthing/lib/db
github.com/syncthing/syncthing/lib/relay/client
github.com/syncthing/syncthing/lib/scanner
github.com/syncthing/syncthing/lib/stats
github.com/syncthing/syncthing/lib/config
github.com/syncthing/syncthing/cmd/syncthing/decrypt
github.com/syncthing/syncthing/lib/nat
github.com/syncthing/syncthing/lib/stun
github.com/syncthing/syncthing/lib/discover
github.com/syncthing/syncthing/lib/watchaggregator
github.com/syncthing/syncthing/cmd/syncthing/cli
github.com/syncthing/syncthing/lib/versioner
github.com/syncthing/syncthing/lib/pmp
github.com/syncthing/syncthing/lib/upnp
github.com/syncthing/syncthing/lib/connections
github.com/syncthing/syncthing/lib/ur
github.com/syncthing/syncthing/lib/model
github.com/syncthing/syncthing/lib/api
github.com/syncthing/syncthing/lib/syncthing
github.com/syncthing/syncthing/cmd/syncthing/generate
github.com/syncthing/syncthing/cmd/syncthing
*** Finished build for x86_64
All builds finished

and running it on the Android 13 x86_64 emulator from Android studio gives a stable-running SyncthingNative:

image

Sorry to bother you again, but now I need your expertise as I'm not familiar with those go things. Which of the code lines could cause the problem only relevant to x86_64 builds ? Maybe we can report a fix/PR to the inotify-go mod maintainers together?

	epes := make([]unix.EpollEvent, 1)
	fd := atomic.LoadInt32(&i.fd)
	for {
		switch _, err := unix.EpollWait(i.epfd, epes, -1); err {
		case nil:
			switch epes[0].Fd {
			case fd:
				esch <- i.read()
				epes[0].Fd = 0
			case int32(i.pipefd[0]):
				i.Lock()
				defer i.Unlock()
				if err = unix.Close(int(fd)); err != nil && err != unix.EINTR {
					panic("notify: close(2) error " + err.Error())
				}
				atomic.StoreInt32(&i.fd, invalidDescriptor)
				if err = i.epollclose(); err != nil && err != unix.EINTR {
					panic("notify: epollclose error " + err.Error())
				}
				close(esch)
				return
			}
		case unix.EINTR:
			continue
		default: // We should never reach this line.
			panic("notify: epoll_wait(2) error " + err.Error())
		}
	}

@Catfriend1
Copy link
Owner Author

In another issue I've found by googling around, they did update the fsnotify module to solve the same problem: silenceper/gowatch#44

@Catfriend1
Copy link
Owner Author

@Catfriend1
Copy link
Owner Author

From Syncthing's code side, this also lets SyncthingNative run on x86_64:
In "...\syncthing-android\syncthing\src\github.com\syncthing\syncthing\lib\fs\basicfs_watch.go"
image

@Catfriend1
Copy link
Owner Author

Even better:
image
image

@Catfriend1
Copy link
Owner Author

Catfriend1 commented Dec 7, 2022

@imsodin I've found and verified a workaround by disabling the missing syscall on android-amd64 for both Chromebook and the Android emulator (I would really prefer to use the emulator to continue my work on the fork). Any chance that you could be so kind to assist me getting this upstream? It would also help the official Syncthing Android wrapper as currently AVD >= 12 and Chromebook support is broken and would then be functional without inotify watches.

Here's the tested change for Syncthing(Native): In "...\syncthing-android\syncthing\src\github.com\syncthing\syncthing\lib\fs\basicfs_watch.go"

import (
	"context"
	"errors"
	"runtime"

	"github.com/syncthing/notify"
)

(I've added "runtime")

and in "func (f *BasicFilesystem) Watch" I've added the if (runtime.GOOS ...)

	if (runtime.GOOS == "android" && runtime.GOARCH == "amd64") {
		err = errors.New("failed to setup inotify handler, see https://github.com/Catfriend1/syncthing-android/issues/583")
		return nil, nil, err
	}

after this code (lines 34 to 37):

	eventMask := subEventMask
	if !ignorePerms {
		eventMask |= permEventMask
	}

@Catfriend1
Copy link
Owner Author

@ivysrono Does the recent v1.23.0.0 release of Syncthing-Fork work on your device?

@ivysrono
Copy link

Yes.

@ivysrono Does the recent v1.23.0.0 release of Syncthing-Fork work on your device?

@Catfriend1
Copy link
Owner Author

@ivysrono Then, I really wonder why Syncthing-Android (official) doesn't run. It's the same native and it contains a patch to make it work on this OS arch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants