question: .proto packaging / repository layout

[kudla]

Hi
Trying to summarize the question is something like
How to write a pure reusable distributed proto packages with arbitrary depth of transitive dependencies.
E.g A, B->A, C->(A,B) : where A,B,C - are all separated repos with a proto declarations. The same time in a way it works for google/protobuf/empty.proto.

I'm not sure how better to structure the entire question. Instead there's a batch of tiny ones that looks contradicting the subject intention or just confusing the hole picture vision.

1. All the examples shows a package naming as "foo.bar.baz". Still google/protobuf/empty.proto is imported as some url(fs) path.
2. When referring import "some/other/mod.proto" does it really mean the referring (linking) on an initial photo-source level or at the compiled to target language one. As we can see google/protobuf/empty.proto as a source reference. Still the result compilation refers the real golang "google.golang.org/protobuf/types/known/emptypb" module.
3. The most resources say the only (best) composition for protos is a subtree/submodule linking
   3.1 So that how to keep code clean and consistent that case. As I can see every level of repo reference adds additional fs tree nesting to a target
   E.g.

repoA
   a.proto

repoB
   A(->repoA)
     a.proto
   B
     b.proto

repoC
  A(->repoA)
       a.proto
  B(->repoB)
     A(->repoA)
       a.proto
     B
       b.proto
   C
      c.proto

3.2 As well. How to avoid a code duplication in case of C->(B, A) if B is just a black box that doesn't leak it's dependancies and to compose all the reps within C as flat set.
3.3 Anyway in case of google/protobuf/empty.proto we don't subtree/fetch/declare whatever manually at all. So how to achieve. the same with custom repos.
4. How to make it even fluent and language agnostic in different packaging managers, e.g. go mod, npm etc. as the mentioned google/protobuf/empty.proto does.

Thank you

[stapelberg]

Hey @kudla

Your question is long and has many parts, not all of which I understand.

I’ll try to help as best as I can:

1. The package name (e.g. google.protobuf) is separate from the file system path (e.g. google/protobuf/empty.proto) of a .proto file. While they are separate, a common naming strategy is for the package to match the file system directory (as seen here).

2. The path google/protobuf/empty.proto refers to a “well-known proto”, i.e. one which is shipped with protoc itself.

3. If, during proto compilation (running protoc), you want to import other .proto files, you need to adjust protoc’s import search path by using the -I flag.

   • The well-known protos are found the same way: the import search path by default points to the location where the well-known protos can be found, which is the include/ directory next to the bin/ directory in which protoc lives:

   % unzip -l protoc-29.1-linux-x86_64.zip 
   Archive:  protoc-29.1-linux-x86_64.zip
     Length      Date    Time    Name
   ---------  ---------- -----   ----
           0  1980-01-01 00:00   bin/
     9623664  1980-01-01 00:00   bin/protoc
           0  1980-01-01 00:00   include/
           0  1980-01-01 00:00   include/google/
           0  1980-01-01 00:00   include/google/protobuf/
        6154  1980-01-01 00:00   include/google/protobuf/any.proto
        7729  1980-01-01 00:00   include/google/protobuf/api.proto
           0  1980-01-01 00:00   include/google/protobuf/compiler/
        8556  1980-01-01 00:00   include/google/protobuf/compiler/plugin.proto
        2185  1980-01-01 00:00   include/google/protobuf/cpp_features.proto
       52531  1980-01-01 00:00   include/google/protobuf/descriptor.proto
        4892  1980-01-01 00:00   include/google/protobuf/duration.proto
        2363  1980-01-01 00:00   include/google/protobuf/empty.proto
   […]
   

4. For repository b to work with a service that is implemented in repository a, it is typically sufficient to import the generated protobuf Go code (.pb.go files). You typically don’t need the .proto files from a in b.

5. (I don’t know enough about npm or other package managers to say anything about them.)