Skip to content
This repository has been archived by the owner on Mar 1, 2022. It is now read-only.

node: compile protos into JSON #678

Merged
merged 4 commits into from
Jul 31, 2019
Merged

node: compile protos into JSON #678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
aaaa927
node: compile protos to JSON
alexander-fenster Jul 30, 2019
b14115c
fix golden tests
alexander-fenster Jul 31, 2019
88b9255
set the minimum version for google-gax
alexander-fenster Jul 31, 2019
81b4ab4
Merge branch 'master' into nodejs-compile-protos
alexander-fenster Jul 31, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 13 additions & 8 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,6 @@ RUN apt-get update \
python3-pip \
# Java
openjdk-8-jdk-headless \
# NodeJS
# This installs Node 4 on Ubuntu 16.04.
nodejs \
npm \
# Ruby
ruby \
ruby-dev \
Expand All @@ -62,8 +58,21 @@ RUN apt-get update \
libunwind8 \
libuuid1 \
zlib1g \
xz-utils \
&& rm -rf /var/lib/apt/lists/*

# Download and unpack Node.js v10
RUN mkdir -p /opt \
&& curl -L https://nodejs.org/dist/v10.16.0/node-v10.16.0-linux-x64.tar.xz -o /opt/node-v10.16.0-linux-x64.tar.xz \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Little worried about how we keep the version of nodejs you're running here up to date.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, first of all, it does not make it any worse because previously it was Node 4 from Ubuntu :) Then, this version of Node.js is only used for compiling protos. I suggest we leave both versions (Node.js and google-gax) hardcoded here for now (same way as we do for other stuff in this Dockerfile such as protoc version, etc.) and update as needed.

Actually, we'll get rid of artman soon (I hope so) so who cares?

&& tar -C /opt -xJf /opt/node-v10.16.0-linux-x64.tar.xz \
&& rm -f /opt/node-v10.16.0-linux-x64.tar.xz
ENV PATH /opt/node-v10.16.0-linux-x64/bin:$PATH

# Install google-gax for Node.js
RUN npm install -g google-gax@^1.2.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh bother. What's going to keep that version up to date?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well... semver ^1.2.1 will allow everything up to 2.0, and it actually prevents me from possible breaking changes in google-gax. It's pretty much the same way as we do in package.json, isn't it?

# Run the compileProtos script for the first time to download runtime dependencies; ignore exit code
RUN compileProtos || true

# Install all required protoc versions, and install protobuf Python package.
ADD install_protoc.sh /
RUN bash install_protoc.sh
Expand All @@ -78,10 +87,6 @@ RUN curl -L https://www.nuget.org/api/v2/package/Grpc.Tools/1.17.1 -o temp.zip \
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME

# Ubuntu apt uses "nodejs" as the executable, but everything else expects
# the executable to be spelled "node".
RUN ln -s /usr/bin/nodejs /usr/local/bin/node

# Install Go.
RUN mkdir -p /golang \
&& cd /golang \
Expand Down
24 changes: 23 additions & 1 deletion artman/tasks/protoc_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"""Tasks related to protoc"""

import io
import json
import os
import re
from ruamel import yaml
Expand Down Expand Up @@ -250,18 +251,39 @@ def execute(self, grpc_code_dir):


class NodeJsProtoCopyTask(task_base.TaskBase):
"""Copies the .proto files into the gapic_code_dir/proto directory.
"""Copies the .proto files into the gapic_code_dir/proto directory
and compiles these proto files to protobufjs JSON.
"""
def execute(self, gapic_code_dir, src_proto_path, excluded_proto_path=[]):
final_output_dir = os.path.join(gapic_code_dir, 'protos')
src_dir = os.path.join(gapic_code_dir, 'src')
proto_files = []
for proto_path in src_proto_path:
index = protoc_utils.find_google_dir_index(proto_path)
for src_proto_file in protoc_utils.find_protos(
[proto_path], excluded_proto_path):
relative_proto_file = src_proto_file[index:]
proto_files.append(relative_proto_file)
dst_proto_file = os.path.join(
final_output_dir, relative_proto_file)
dst_proto_dir = os.path.dirname(dst_proto_file)
if not os.path.exists(dst_proto_dir):
self.exec_command(['mkdir', '-p', dst_proto_dir])
self.exec_command(['cp', src_proto_file, dst_proto_file])
# Making transition to JSON protos simple:
# If this library has the list of protos for us, use it.
# Otherwise, just do its job and build this list.
# When all libraries start generating list of protos, the following code
# (up to the compileProtos execution) will be removed.
source_files = protoc_utils.list_files_recursive(src_dir)
proto_json_files = [filename for filename in source_files if re.match('_proto_list\\.json$', filename)]
if len(proto_json_files) == 0:
proto_list_file = os.path.join(src_dir, 'service_proto_list.json')
proto_files_relative = ['../protos/' + filename for filename in proto_files]
with open(proto_list_file, 'w') as pf:
pf.write(json.dumps(proto_files_relative))
# Execute compileProtos from Docker image (a part of from google-gax)
cwd = os.getcwd()
os.chdir(gapic_code_dir)
self.exec_command(['compileProtos', './src'])
os.chdir(cwd)
2 changes: 2 additions & 0 deletions test/golden/library_example.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,9 @@
/js/library_example-v1/README.md
/js/library_example-v1/package.json
/js/library_example-v1/protos/google/example/library/v1/library.proto
/js/library_example-v1/protos/protos.json
/js/library_example-v1/src/index.js
/js/library_example-v1/src/service_proto_list.json
/js/library_example-v1/src/v1/doc/google/example/library/v1/doc_library.js
/js/library_example-v1/src/v1/doc/google/protobuf/doc_empty.js
/js/library_example-v1/src/v1/index.js
Expand Down