Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: improve logging with additional test coverage #1937

Merged
merged 4 commits into from
Sep 25, 2023
Merged

Conversation

annafang-google
Copy link
Contributor

Let proxy continue to run even instances specified are inactive.
Tested these scenarios:

  • inactive instances without TCP or Unix socket options
  • inactive instances with TCP port
  • inactive instances with Unix socket

}
}
return nil, fmt.Errorf("[%v] Unable to mount socket: %v", inst.Name, err)
fmt.Errorf("[%v] Unable to mount socket: %v", inst.Name, err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to either log this error, or return it. Right now, this is creating an error value and discarding it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is only one instance, should we still fail the proxy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced it with logger.Errorf(). Thanks!
Considering the scenario where this one instance was accidentally stopped, having the proxy run will save a minor effort of restarting proxy after the instance is activated. It's not a huge saving, but the behaviour could be consistent.

internal/proxy/proxy.go Show resolved Hide resolved
internal/proxy/proxy.go Show resolved Hide resolved
internal/proxy/proxy.go Show resolved Hide resolved
@@ -306,6 +309,39 @@ func TestClientInitialization(t *testing.T) {
filepath.Join(testUnixSocketPathPg),
},
},
{
desc: "without TCP port or unix socket for non functional instance",
in: &proxy.Config{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test below is build around wantTCPAddrs or wantUnixAddrs. So as written, this test will never fail. To ensure we have the behavior we want here, we should add a functioning socket with a non functioning one and ensure we get what we expect with wantTCPAddrs or wantUnixAddrs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fake dialer's EngineVersion() added a fakeserver case to mimic the failed instance case. EngineVersion() call will fail in the NewClient() call. What the test does is to make sure that NewClient() will proceed upon the failure of EngineVersion().

},
},
},
wantUnixAddrs: []string{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the fake instance result in a Unix socket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of the auto-reconnecting the instance, it might be good to keep the socket?

@enocom
Copy link
Member

enocom commented Sep 1, 2023

A couple of high level thoughts:

  1. It is possible to specify a port to allow the Proxy to run with inactive instances. In fact, we recommend production workloads do this anyway to speed up the initial startup (and avoid waiting on the metadata call)
  2. It is not possible to do this with Unix sockets presently because we must know if an instance is postgres or not, and right now the Go Connector will fail if an instance isn't up (source). We could split that call into two parts with some work, but perhaps the value of this is less high.
  3. It's probably best to just skip failed Unix sockets with an error message and call it a day given we already support this feature otherwise.

WDYT?

@annafang-google annafang-google force-pushed the non-fatal branch 3 times, most recently from cff7808 to db06ac9 Compare September 5, 2023 15:24
@annafang-google
Copy link
Contributor Author

Makes sense. Updated the PR that fails the unix path if the instance is not active.

@@ -747,6 +747,14 @@ type socketMount struct {
dialOpts []cloudsqlconn.DialOption
}

func getNetworkType(conf *Config, inst InstanceConnConfig) string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this networkType. Cf. https://go.dev/doc/effective_go#Getters

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thanks!

}

l.Infof("[%s] Listening on %s", inst.Name, m.Addr())
mnts = append(mnts, m)
}
c.mnts = mnts
return c, nil
if len(mnts) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's swap this with the error below, so the error path is indented.

https://github.com/golang/go/wiki/CodeReviewComments#indent-error-flow

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thanks!

},
}

unixTcs := []struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we move this one test case into a dedicated test below? e.g., TestProxyInitializationWithFailedUnixSocket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to make the test more explicit! Fixed

@@ -801,6 +807,8 @@ func (c *Client) newSocketMount(ctx context.Context, conf *Config, pc *portConfi

address, err = newUnixSocketMount(inst, conf.UnixSocket, strings.HasPrefix(version, "POSTGRES"))
if err != nil {
c.logger.Errorf("could not mount unix socket %q for %q: %v",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's consolidate all logging up above in NewClient.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that we're not doing that consistently above. We should confirm that we're not logging this error twice, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, removed the NewClient logs and kept logs at the newSocketMount level.

internal/proxy/proxy.go Show resolved Hide resolved
internal/proxy/proxy.go Outdated Show resolved Hide resolved
internal/proxy/proxy_test.go Show resolved Hide resolved
@annafang-google annafang-google force-pushed the non-fatal branch 3 times, most recently from 722c84a to 30238d6 Compare September 6, 2023 14:45
Replaced fmt.Errorf with logger.Errorf for instance failure

Fail the inactive instance with unix path

Fixed lint error

Added inactive instance test with unix path

Replace getNetworkType with networkType

Fixed the error path indentation

Make the unix path failure test a seperate func

Fail the proxy binary if the tcp socket argument is wrong
Copy link
Member

@enocom enocom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Key point: let's add support here for skipped over failed Unix sockets. Otherwise, we should require users specify a port for TCP sockets to avoid adding any additional complexity here.

internal/proxy/proxy.go Outdated Show resolved Hide resolved
switch err.(type) {
// this is not the inactive instance case. Fail the proxy
// so that the user could fix the proxy command error.
case *net.OpError:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what cases does this error get returned? Could we add a comment clarifying that?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this isn't Unix related, I think we shouldn't add it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the comments. This is to handle the case where listen call could fail.

@@ -801,6 +807,8 @@ func (c *Client) newSocketMount(ctx context.Context, conf *Config, pc *portConfi

address, err = newUnixSocketMount(inst, conf.UnixSocket, strings.HasPrefix(version, "POSTGRES"))
if err != nil {
c.logger.Errorf("could not mount unix socket %q for %q: %v",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that we're not doing that consistently above. We should confirm that we're not logging this error twice, though.

internal/proxy/proxy.go Show resolved Hide resolved
@enocom enocom self-requested a review September 13, 2023 21:37
@enocom enocom assigned enocom and unassigned jackwotherspoon Sep 19, 2023
for _, tc := range tcs {
t.Run(tc.desc, func(t *testing.T) {
_, err := proxy.NewClient(ctx, &fakeDialer{}, testLogger, tc.in)
if tc.wantSuccess != (err == nil) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be:

if tc.wantSuccess && err != nil {
// ...
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the proposed change, if tc.wantSuccess is false (we want NewClient() to fail), the branch will not be executed regardless what the err is. This will miss reporting the failing case when err is nil and we want NewClient() to fail.

mErr := m.Close()
if mErr != nil {
l.Errorf("failed to close mount: %v", mErr)
switch err.(type) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of switching on error type, could we just continue if the socket type is a Unix socket (like what we had before)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will miss the case when the type is TCP socket and there is no active instance. Correct me if I'm wrong, my understanding from the discussion offline:

  • Inactive instance with unix socket should fail
  • Inactive instance with TCP socket with no port number configured should fail.

@annafang-google
Copy link
Contributor Author

annafang-google commented Sep 22, 2023

As discussed offline, keep the logic to fail the proxy if the proxy is unable to connect to inactive instances. Retain the logging to keep it consistent.
The recommended way to keep the proxy running is to specify the port number in the configuration for TCP case. The unix socket case however, because the proxy has no way to identify if the db is postgres (for inactive instance), the code will fail the proxy in this case.

@enocom enocom changed the title fix: let proxy continue to run with inactive instance(s) (#1854) chore: improve logging with additional test coverage Sep 22, 2023
@enocom enocom self-requested a review September 22, 2023 21:06
@annafang-google annafang-google merged commit d88b907 into main Sep 25, 2023
12 checks passed
@annafang-google annafang-google deleted the non-fatal branch September 25, 2023 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants