Deprecating relative path and forcing full path doesn't appear to be compatible with Go Modules

[halverneus]

What version of protobuf and what language are you using?
protoc 3.11.3 using Go protoc-gen-go

What did you do?

main.go

package main

//go:generate protoc example.proto --go_out=plugins=grpc:$GOPATH/src

import (
	"fmt"

	"somewhere.com/path/to/mine"
)

func main() {
	v := mine.Empty{}
	fmt.Println(v)
}

example.proto

syntax = "proto3";

package mine;
option go_package = "somewhere.com/path/to/mine";

message Empty {}

Steps to reproduce:

export GO111MODULE=on
go generate ./...
go run main.go

What did you expect to see?
I expected the program to compile and run. This change has become important because of this warning message from protoc-gen-go:

WARNING: Deprecated use of 'go_package' option without a full import path in "files.proto", please specify:

I can find no way to use a full import path that works with Go Modules.

What did you see instead?
The protobuf file is generated into the GOPATH, but Go Modules doesn't look in the GOPATH for the built module.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment?

• go version go1.14.2 linux/amd64
• libprotoc 3.11.3
• protoc-gen-go installed with: go get github.com/golang/protobuf/protoc-gen-go

[neild]

The protobuf file is generated into the GOPATH, but Go Modules doesn't look in the GOPATH for the built module.

Your protoc invocation is requesting that output be placed in $GOPATH/src:

//go:generate protoc example.proto --go_out=plugins=grpc:$GOPATH/src

If you want the output somewhere else, you'll need to tell protoc where that is.

[halverneus]

So, where should it go to work with Go Modules?

…

On Thu, Apr 23, 2020, 3:14 PM Damien Neil ***@***.***> wrote: The protobuf file is generated into the GOPATH, but Go Modules doesn't look in the GOPATH for the built module. Your protoc invocation is requesting that output be placed in $GOPATH/src: //go:generate protoc example.proto --go_out=plugins=grpc:*$GOPATH/src* If you want the output somewhere else, you'll need to tell protoc where that is. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQDXNFZQIZ7I73IR3SFEWDROC4TJANCNFSM4MPRU4AQ> .

[dsnet]

What's the name of your module?

Assuming you have protoc-gen-go at v1.4.0 or higher, something like:

protoc --go_out=module=example.com/module/root:. source.proto

See the release notes for details.

[halverneus]

I'm about to try it out, but will that work for grpc services? In this example, the module would be "somewhere.com/path/to/mine" (as listed in the example.proto go_package option that forcing adoption). Everything I've read in states that using relative paths or leaving off the go_package is being deprecated and I'm trying to figure out how to generate GRPC client and server code using Go Modules once the legacy method is not supported.

Simply, today I learned that the way my company builds GRPC services is being deprecated, the closest solution I could find after searching online all day was to do the "--go_out=plugins=grpc:$GOPATH/src" but that method does not work with Go Modules and I cannot find documentation anywhere that describes how to use this solution with Go Modules (unless I create entire repos that host generated code from GRPC files, but that violates "best practices" and adds needless complexity).

[dsnet]

I'm about to try it out, but will that work for grpc services?

It should for [email protected], but not for [email protected] and above. We're in the process of handing off ownership of the grpc plugin to the grpc-go project (see #903). It's been hosted as part of protoc-gen-go primarily for historical reasons.

[halverneus]

I'm afraid I just ran out of time, today. I'll try it out in the morning. Thank you very much!

[halverneus]

This method doesn't seem to work, either. The generated code just appears in the same directory that the "go:generate" command exists and cannot be imported. I also tried moving the full path into the root of the application and it was still not found. Effectively, the file structure ended up looking like: /home/me/code/somewhere.com/path/to/project/somewhere.com/path/to/mine and still doesn't seem to work with Go Modules. In the example code provided (which doesn't use GRPC, but still show the same problem) I changed the code to look like:

package main

//go:generate protoc example.proto --go_out=module=somewhere.com/path/to/mine:.

import (
	"fmt"

	"somewhere.com/path/to/mine"
)

func main() {
	v := mine.Empty{}
	fmt.Println(v)
}

I can confirm that I'm using v1.4.0. (Actually after realizing that I wasn't updated to the latest, the difference is that the "example.pb.go" appears in the root of the project, but is still not imported.) What makes this difficult is in the actual projects, the code generated from the *.proto file either needs to be placed in a different package path per project (which doesn't seem possible with the go_package option (unless that is just a suggestion that is overridden by the --go_out flag)) or needs to be in its own package, but there is consensus that we do not want to commit generated code to a package. Does that make sense?

[halverneus]

Realizing that I was experiencing a version discrepancy between my host machine and my Dockerfile (host was running an older protoc-gen-go that was incorrectly handling 'option go_package=".;mine"' by creating a package "__mine"), as long as the ".;*" syntax continues to be supported, I don't have an issue. If that syntax is going away in the future, then I'm not seeing a path forward.

[dsnet]

This is the first time I have seen:

option go_package = ".;mine";

What is the "." in the first segment supposed to mean?

[halverneus]

If you use:

option go_package = "mine";

protoc-gen-go warns that it should be ".;mine". My problem arose when I got that warning while building with Docker, then tested on my machine (version <1.4.0) and ended up with packages name "__mine". I have no idea what the syntax means and it makes no sense to me (and I'm not a fan of "magic syntax"), but if it is going to work like that going forward, I can live with it (though it would be nice for that to be documented somewhere online).

[dsnet]

protoc-gen-go warns that it should be ".;mine".

That seems to be a bug with protoc-gen-go's warnings.

The go_package option should specify the full package path for the generated Go package associated with this .proto file. See https://developers.google.com/protocol-buffers/docs/reference/go-generated#invocation for details.

[dsnet]

https://go-review.googlesource.com/c/protobuf/+/232338 adjusts the warning provided by protoc-gen-go to not suggest a poor option if it couldn't determine the full Go package path.

[halverneus]

If the option for a relative path is removed, then I'm back to the original issue for which this was created: I don't see a way to us this with Go Modules.

[dsnet]

Can you explain how full go package paths is incompatible with Go modules?

[halverneus]

You can use the example code and comments above to demonstrate, but in the company I work for (and the company I worked at before this one) we are not allowed to commit generated code. We have a *.proto file that is maintained in a documentation repository that can be used by multiple languages and repositories. We can pull the *.proto service files into our projects and run the code generator on them during our build process. This leaves us two options:

1. We generate the service code using a relative path in the repository where the code is needed. This means the import path is globally different for each project, but it hasn't mattered because it is only required at the executable-building level. It sounds like this method is about to be deprecated.

2. We generate the code using an absolute path that inserts the package path into the Go Modules somehow. Because Go Modules don't pull from the GOPATH, the method I outlined in previous comments doesn't work. Namely, if we want to use this method, then we need for our executable repository (somewhere.com/path/to/repo or somewhere.com/path/to/other) to generate the Go code for the absolute path that can work across all repositories (somewhere.com/path/to/generated/code) but be made available using Go Modules. Either this isn't possible, yet, or no one has answered this, yet. Does that make sense?

It is possible to NOT use Go Modules and use the method outlined above (Go Modules appear to be the future and appear to be the best method for dependency management) OR to generate code and commit it as a separate go-gettable repository (not allowed at any company I've worked for as it violates many documented best-practices).

[dsnet]

We generate the code using an absolute path that inserts the package path into the Go Modules somehow.

I don't know if you're aware, but the module flag for protoc-gen-go seems to be exactly targetted towards the situation you're describing.

[halverneus]

Can you give an example protoc line using the example above? I opened this issue because I spent nearly a day searching online and could not find a method or example documented to accomplish this task. Above, I tried this in one code example by including this line:

//go:generate protoc example.proto --go_out=module=somewhere.com/path/to/mine:.

And it created the *.pb.go file in the local directory, but it couldn't be imported into the application as "somewhere.com/path/to/mine". It appears that I could use the setting option go_package = "somewhere.com/path/to/mine"; and use --go_out=module=somewhere.com/path/to/other/mine and import "somewhere.com/path/to/other/mine", but that seems to make go_package pointless and adds needless confusion (in my case, I don't see the point behind "option go_package" if it doesn't actually do anything with it...).

[dsnet]

//go:generate protoc example.proto --go_out=module=somewhere.com/path/to/mine:.

The module flag is intended to output to the module root directory, not what the source file itself is located.

If you want to output a .pb.go file that is co-located with the source .proto file, then the source_relative flag should be used instead. See https://developers.google.com/protocol-buffers/docs/reference/go-generated#invocation

[halverneus]

The "module" flag makes sense to me. The "option go_package" doesn't. The "option go_package" doesn't seem to be used for anything. In the old version, it got stuffed into the package name at the top of the file, but in the new version just seems to be used to validate that the "package x;" matches the option go_package = "wherever.com/x"; It doesn't seem to have any other purpose.

[dsnet]

The go_package is about protoc-gen-go being able resolve Go package dependencies. While it can be used to specify the Go package name, that is not its primary purpose. If a.proto depends on b.proto, we must know what the relationship between the generated Go packages for these two files. Are they in the same Go package? If no, what is the Go package path used to import the generated Go package for b.proto.

In the old version

In the old version, protoc-gen-go used a very complicated set of heuristics to try and guess the target go package path for a dependency and very often got it wrong. The go_package option is to avoid that fiasco altogether. Rather than try and guess something it rarely gets right. It requires being told that piece of information.

[halverneus]

Makes sense. Thank you for the clarification.