Stop loading proto files dynamically

[alexander-fenster]

Currently, all gRPC Node.js client libraries load proto files dynamically using @grpc/proto-loader (that internally calls protobufjs.loadSync). Since proto files can form a huge dependency tree, it brings some real time complexity to the client constructor, and (more importantly) it blocks things like webpack (#2933).

There are several possible ways to deal with this dynamic load.

1. [proposed option] Compile protos to JSON using pbjs -t json running in a postinstall script, and load this JSON file in the runtime. For Node.js, this JSON format can be read by @grpc/proto-loader without making any changes to it (it's an undocumented feature but since protobufjs supports it, @grpc/proto-loader also supports it). For browsers, we'll do protobuf.root.fromJSON(jsonContent) to load the same JSON file which will be required to become a part of the bundle.

2. Compile proto files to static objects with protoc - this is what other languages do.
   I dislike this option, mostly because the change from protobufjs object to protoc will likely bring some incompatibilities and hard-to-find bugs in the libraries.

3. Compile proto files to static objects with pbjs. Unfortunately, static objects generated by pbjs -t static-module are not supported by @grpc/grpc-js so we can't use this option for now (even though it's a great option for possible browser usage).

So, from my undestanding at this moment, the first option is the only option that would allow us to make the code working both in Node.js without dynamic proto loading, and in browsers (with no gRPC support). I'm going to work on this option and will update this issue when the code is ready to review.

Cc: @JustinBeckwith @bcoe @guybedford

[guybedford]

How important is it to separate the proto files from the main build files in Node.js? Your proposal seems to suggest dynamic JSON loading in Node.js, and a static bundle for the browser. Usually we find the converse - that it is the browser where you want the granular loading of JSON files, and in Node.js where it is better to have everything in one file for their performance.

[alexander-fenster]

@guybedford So the problem here with the underlying library (gRPC) that for now cannot consume the statically created proto object. So we'll try to see how JSON protos performs in both cases (both Node and browser). I can write more words but let me better share some code that will show the idea :) (later this week)

[timconnorz]

Any update here? I'm having issues with webpack & google cloud node client

[JustinBeckwith]

Greetings! If you're having issues, please file a new issue letting us know specifically what you're trying, and what problems you're running into :) I don't think this is the issue you're looking for :P

[alexander-fenster]

@timconnorz Just an update, we've been loading protos from a pre-compiled JSON for a year already, and we have a {fallback: true} option (that also works in Node.js, not just in browsers) that completely removes fs module from the equation. This issue is pretty much outdated so I'm going to close it - feel free to open an issue for the library you're trying to use in its corresponding repository (e.g. googleapis/nodejs-speech, googleapis/nodejs-vision, etc.)

Thank you!