Unify protos, add codegen for all languages, set up for package publi…

…shing

protos/README.md

@@ -0,0 +1,26 @@
+# Protos
+This directory contains the protobuf definitions for all Aptos services. For the sake of simplifying release and minimizing potential version conflicts, we include all protos and code generated from those protos in one place.
+
+If you update the proto definitions in `proto/`, you can regenerate them for all languages by running this script:
+```
+./scripts/build_protos.sh
+```
+
+Make sure you have the following deps installed:
+```bash
+# Install buf
+brew install bufbuild/buf/buf
+
+# For generating Rust code
+cargo install protoc-gen-prost
+cargo install protoc-gen-prost-serde
+cargo install protoc-gen-prost-crate
+cargo install protoc-gen-tonic
+
+# For generating TS code
+pnpm install -g [email protected]
+
+# For generating Python code
+cd python/aptos-indexer-protos
+poetry install
+```

protos/buf.rust.gen.yaml

@@ -0,0 +1,20 @@
+version: v1
+managed:
+  enabled: true
+plugins:
+  - name: prost
+    out: rust/src/pb/
+    opt:
+      - file_descriptor_set
+
+  - name: prost-serde
+    out: rust/src/pb/
+
+  - name: prost-crate
+    out: rust/src/pb/
+    strategy: all
+    opt:
+      - no_features
+
+  - name: tonic
+    out: rust/src/pb/

protos/buf.ts.gen.yaml

@@ -0,0 +1,19 @@
+version: v1
+managed:
+  enabled: true
+plugins:
+  - plugin: buf.build/community/stephenh-ts-proto
+    out: typescript/src
+    opt:
+      # See an explanation of these options here:
+      # https://github.com/stephenh/ts-proto
+      - outputServices=grpc-js
+      - outputIndex=true
+      - esModuleInterop=true
+      - forceLong=bigint
+      - useOptionals=all
+      - addGrpcMetadata=true
+      - useAsyncIterable=true
+      - useMapType=true
+      # https://github.com/stephenh/ts-proto/issues/449
+      - useExactTypes=false

protos/buf.work.yaml

@@ -0,0 +1,3 @@
+version: v1
+directories:
+  - proto

protos/proto/aptos/bigquery_schema/v1/transaction.proto

@@ -0,0 +1,33 @@
+// Copyright © Aptos Foundation
+// SPDX-License-Identifier: Apache-2.0
+
+// Proto2 is required.
+// Current BigQuery runs over proto2, thus optional(nullable)
+// field with default value will be ignored. For example,
+// `int64 value = null` will be translated to 0 under column `value`.
+// To avoid any analytics hassle, proto2 is required here.
+syntax = "proto2";
+
+package aptos.bigquery_schema.transaction.v1;
+
+// Transaction is a simplified representation for the transaction
+// happened on the chain. Mainly built for streaming into BigQuery.
+// It matches with the structure defined for the transaction in Indexer.
+message Transaction {
+    required int64 version = 1;
+    required int64 block_height = 2;
+    required string hash = 3;
+    required string type = 4;
+    optional string payload = 5;
+    required string state_change_hash = 6;
+    required string event_root_hash = 7;
+    optional string state_checkpoint_hash = 8;
+    required uint64 gas_used = 9;
+    required bool success = 10;
+    required string vm_status = 11;
+    required string accumulator_root_hash = 12;
+    required int64 num_events = 13;
+    required int64 num_write_set_changes = 14;
+    required int64 epoch = 15;
+    required int64 inserted_at = 16;
+}

protos/python/.gitignore

@@ -0,0 +1,161 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+myvenv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

protos/python/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Aptos Protos Changelog
+
+All notable changes to the Aptos Protos will be captured in this file. This changelog is written by hand for now.
+
+## Unreleased
+
+## 1.1.2
+- Initial release.

protos/python/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Aptos Indexer Protos
+
+## Regenerating code
+Note: This is only for debugging. For normal uses, just use scripts/build_protos.sh in the parent directory.
+
+Install deps:
+```
+poetry install
+```
+
+Generate code:
+```
+poetry run poe generate
+```
+
+## Publishing
+To publish the package, follow these steps.
+
+First, make sure you have updated the changelog and bumped the package version if necessary.
+
+Configure Poetry with the PyPi credentials:
+```
+poetry config pypi-token.pypi <token>
+```
+
+You can get the token from our credential management system, search for PyPi.
+
+Build and publish:
+```
+poetry publish --build
+```

protos/python/README.md

@@ -0,0 +1,18 @@
+# Aptos Indexer Protos
+
+This repository contains the protobuf definitions for Aptos.
+
+## Usage
+Import generated classes like this:
+```python
+from aptos_protos.aptos.transaction.v1.transaction_pb2 import Transaction
+```
+
+Then use them like this:
+```python
+def parse(transaction: Transaction):
+    # Parse the transaction.
+```
+
+## Contributing
+See [CONTRIBUTING.md](CONTRIBUTING.md) for more information.

protos/python/aptos_protos/__init__.py

@@ -0,0 +1,7 @@
+# The following is required to make it possible to use the generated code as a package:
+# https://github.com/protocolbuffers/protobuf/issues/881#issuecomment-1615919615
+
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))