Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic when starting stanza using a service account without Log Writer role (Google Output) #264

Closed
jsirianni opened this issue Mar 31, 2021 · 2 comments
Labels
bug Something isn't working

Comments

@jsirianni
Copy link
Member

jsirianni commented Mar 31, 2021

Describe the bug

Stanza panics if the service account does not have Logs Writer.

To Reproduce
Steps to reproduce the behavior:

  1. start stanza with a working configuration, send logs to Google
  2. remove the service accounts "Logs Writer" role
  3. wait for stanza error log to indicate that there is a permission issue
  4. stop stanza
  5. start stanza (panic occurs)

Expected behavior

Stanza should start fine, and continue to log permission errors.

Screenshots

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13346ed]

goroutine 1 [running]:
github.com/observiq/stanza/operator/builtin/input/tcp.(*TCPInput).Stop(0xc00057b040, 0xc00000ecff, 0x0)
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/operator/builtin/input/tcp/tcp.go:229 +0x2d
github.com/observiq/stanza/pipeline.(*DirectedPipeline).Stop(0xc00000ea78, 0x0, 0xc003911668)
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/pipeline/directed.go:42 +0x13f
github.com/observiq/stanza/agent.(*LogAgent).Stop.func1()
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/agent/agent.go:36 +0x45
sync.(*Once).doSlow(0xc0004903ec, 0xc0004a1ab0)
	/usr/local/Cellar/go/1.16.2/libexec/src/sync/once.go:68 +0xec
sync.(*Once).Do(...)
	/usr/local/Cellar/go/1.16.2/libexec/src/sync/once.go:59
github.com/observiq/stanza/agent.(*LogAgent).Stop(0xc0004903c0, 0x0, 0x0)
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/agent/agent.go:35 +0x7d
main.(*AgentService).Stop(0xc0005ace10, 0x214c348, 0xc0005fb3b0, 0x7, 0xc0034bfa68)
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/cmd/stanza/service.go:36 +0xbd
github.com/kardianos/service.(*systemd).Run(0xc0005fb3b0, 0xc000490480, 0xc000134090)
	/Users/jsirianni/git/go/pkg/mod/github.com/kardianos/[email protected]/service_systemd_linux.go:229 +0xe9
main.runRoot(0xc000138c80, 0xc0000b7700, 0x0, 0x4, 0xc000134090)
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/cmd/stanza/root.go:111 +0x453
main.NewRootCmd.func1(0xc000138c80, 0xc0000b7700, 0x0, 0x4)
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/cmd/stanza/root.go:47 +0x52
github.com/spf13/cobra.(*Command).execute(0xc000138c80, 0xc000088060, 0x4, 0x4, 0xc000138c80, 0xc000088060)
	/Users/jsirianni/git/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000138c80, 0x0, 0x2e27998, 0xc00008a058)
	/Users/jsirianni/git/go/pkg/mod/github.com/spf13/[email protected]/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/jsirianni/git/go/pkg/mod/github.com/spf13/[email protected]/command.go:897
main.main()
	/Users/jsirianni/git/go/src/github.com/observiq/stanza/cmd/stanza/main.go:9 +0x2b

Environment:

  • OS: Ubuntu 18.04 LTS
  • Stanza Version: v0.13.16
@jsirianni jsirianni added the bug Something isn't working label Mar 31, 2021
@jsirianni
Copy link
Member Author

Looking into this further, it seems to happen when

  1. There are entires buffered in the database
  2. Stanza starts, fails to init Google output because test connection fails
  3. Input operators are told to stop because an operator (google) failed. This causes the panic.

Not sure why not.

@jsirianni jsirianni mentioned this issue Jun 16, 2021
4 tasks
@jsirianni
Copy link
Member Author

Resolved here #331

This can happen if an input operator tries to close something (such as a connection) when it failed to start, causing the panic. Unfortunately I forgot to post the config used in this specific example. I do not believe this is a Google Output issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant