Support package prefixes

[lally]

Request
Add a flag to proto-lens-protoc to support a package prefix. E.g, a proto Qux.proto normally generating to Proto.Qux, with --proto-prefix Foo.Bar becomes Proto.Foo.Bar.Qux.

Motivation
Hi, I'm using protos from our large codebase in another language. Their build process runs protoc in different subdirectories, and they often have relative imports. I've been slowly hacking up a local copy of proto-lens-setup to find the different protos in that codebase (outside, safely, in my haskell package), but these relative imports are killing me. Different subdirectories act as roots for their protos' imports, and that ends up being quite messy.

I can process them separately in my hacked-up proto-lens-setup, but the resulting module names are a problem.

Example
Packages in foo/bar/baz/{a,b,c} will refer to one another as import a/pants, import b/jacket, import c/socks, etc.

But other proto files will refer to them from the codebase-root import foo/bar/baz/a/pants. So, I'd like to be able to invoke protoc on foo/bar/baz/a/pants.proto in directory foo/bar/baz with --package-prefix Foo.Bar.Baz. That way when a/pants.proto does an import b/jacket.proto, it'll resolve correctly due to its cwd, but generate a file in package Proto.Foo.Bar.Baz.A.Pants. When another proto (which I'll invoke protoc on separately) does an import foo/bar/baz/a/pants, it'll resolve to the correct generated file.

[blackgnezdo]

Do you have any prior art from other languages we can try to follow?

[lally]

I think base_namespace of the c# proto compiler partially fits: https://developers.google.com/protocol-buffers/docs/reference/csharp-generated In that it eats up the path-prefix of the generated file.

Alternatively, I think one could think of this as making the built-in "Proto" prefix configurable. I'd change it to something like Proto.Foo.Bar when I need a prefix.

[blackgnezdo]

@judah do you have any opinion about this kind of knob? It is of little use in monorepo/bazel cases where everything is always built from the top of the tree workspace directory.

[judah]

This situation seems a bit odd to me. If I'm understanding correctly, you have something like this in your existing codebase:

• foo/a/pants.proto
• foo/b/jacket.proto contains import "a/pants.proto"
• bar/baz.proto contains import "foo/a/jacket.proto" and import "foo/b/socks.proto"

And you have another existing language (e.g. C#, C++, Go, ...) that you're already using to process these files. If so, can you explain what its generated module names / imports look like (or whatever the equivalent is for that language)? Are they generated by options in the .proto files themselves, like go_package or java_package? Otherwise it's not clear to me how those inconsistent proto imports would turn into consistent module names.

It might also be tricky when you want to generate bar/file.proto into Proto.Bar.Baz. protoc needs to be able to find all transitive proto files, and you still need to satisfy the "a/pants.proto" import. so e.g. for the above files I needed to pass both -I. and -Ifoo:

protoc --plugin=protoc-gen-haskell=$PROTOC_GEN_HASKELL \
  --haskell_out=out some/other/file.proto -I. -Ifoo

Maybe that would be ok with your setup; I'm not sure.

[lally]

Hi, yes, we option go_package and the package name is set to the path from ~/go/src. So in the example, we'd see foo/a/pants.proto have option go_package = "github.com/us/repo/foo/a/pants". The non-relative imports are actually import "github.com/us/repo/foo/a/pants.proto".

Imports are done either with a fully-qualified path (from ~/go/src, as above) or relatively from some parent path. The latter parent paths (there are many directories of protos in this codebase) are determined by the build process for the different systems that use these protos.

What I've hacked up in my version of proto-lens-setup is a config file that stores the path of the root (~/go/src). I'd like for it to also hold the paths inside of it that need relative path processing. In the running example, that would include ~/go/src/github.com/us/repo/foo, and would match any proto under that directory. Then it would compile protos in that directory with -I~/go/src -I., --proto-prefix=foo.

I see your point about the dependencies having to resolve. Hmm. I was under the impression that the import statements would just result in Haskell import statements, that there wasn't any reading of the imported code again. I say this because when I tried hacking things together with symlinks, the code would generate but wouldn't compile, because it would fail import Proto.A.Pants. My reasoning, based on that belief, is that if I can get the individual Haskell files generated in the right places with the right names, that everything should resolve correctly. But that requires that the imports are resolved against the protoc include path, which I'm not sure is happening.

Otherwise, some sort of smart resolution of the import statements before processing each proto file may be a better idea. So far, I've been hacking up a local branch of the repo containing the protos and making all the import statements non-relative. I'm not committing that back up for two reasons (a) it might make other programmers avoid using relative imports, which is completely unfair, (b) it's showing weakness about the Haskell ecosystem.

Making the compiler do that would make life much simpler. I don't know if that's a workable solution to generalize outside of my org, but I bet a lot of folks trying to get some Haskell working in their non-Haskell codebases would find this useful. It would require that I pass in a --relative-parent=foo to every invocation of protoc, to make sure that when I compile bar/baz.proto, its import of foo/a/jacket.proto would cause it to interpret jacket.proto's relative imports correctly. That could get a bit unweildly for cases where there are a lot of these directories, but I'm running out of alternative ideas.