Dealing with protobufs in multiple packages

[mattberkowitz]

In our protobuf repo we are currently using subpackages to organize our definitions (here's a simple example https://github.com/mattberkowitz/go-proto-test), however we're having trouble compiling them into go:

• If we try to compile all the definitions at once we get an "inconsistent package names" error
• If we compile the packages separately then we get an import cycle error when we try and use them in go
• If we change all the files to have the same go_package option then messages in the root package that have conflicting names with messages in the sub package get overwritten

We've considered renaming the messages in the sub package to avoid conflicts (which should allow us to use the third option above) but would prefer to keep the names as they are rather than adding unnecessary prefixes.

An ideal solution would be to be able to do something like append the subpackage name to the class when compiling into a single go package (ie sub.TypeOne would become subTypeOne) although we've not found an option to accomplish anything like that.

Any other ideas on how we might solve this problem would be greatly appreciated!

[zellyn]

For the Go plugin, you have to compile them one package at a time. You might find our tool helpful with that.

The reason for import cycles is usually that Go has stricter cycle requirements than Protos do: Protos disallow only an import cycle of direct file-level imports. Go disallows package-level circular imports. So if A/red.proto imports B/red.proto and B/blue.proto import A/blue.proto, protoc will be happy, but Go will not. The solution is distinct subgroups of protos into different packages (eg. red/ and blue/) in one of two ways: if you are able, you can simply move them. If other things and languages already use them and you can't break them, move just the Go packages, using a go_package statement.

Having Go-language package import circularities usually is a sign that your proto import structure is a bit confused anyway, so the reorganization will probably be clarifying.

[alecthomas]

Also see #39

[neild]

I believe @zellyn's answer covers this exactly. You need to come up with some division of source .proto files into Go packages that doesn't include any import cycles. In the worst case, you should always be able to put each source file into a different Go package and have things work, since protos disallow file-level import cycles.