Auto-gen Python Exchange wrapper (#1919)

* Rename existing wrapper, to match contract name * base contract: make member var public * json_schemas.py: stop storing copies of schemas! * .gitignore generated erc20_token.py wrapper * json schemas: allow uppercase digits in address * existing exchange wrapper: re-order methods to match method order in Solidity contract, to reduce noise in upcoming diffs of newly generated code vs. old manually-written code. * existing exchange wrapper: rename method params To match contract method param names * existing exchange wrapper: remove redundant member * existing exchange wrapper: make signatures bytes Not strings. * abi-gen/test-cli: show context on diff failure * abi-gen-templates/Py: fix broken event interface Previous changes had removed the `token_address` parameter from all generated methods, but this instance was missed because there weren't tests/examples using events for the first contract for which wrappers were generated (ERC20Token). * abi-gen: remove unused method parameters * abi-gen: convert Py method params to snake case * abi-gen: rewrite Python tuple handling * python-generated-wrappers: include Exchange * abi-gen-templates/Py: easy linter fixes * abi-gen-templates/Py: satisfy docstring linters * abi-gen-templates/Py: normalize bytes before use * contract_wrappers.py: replace Exchange w/generated * contract_wrappers.py: rm manually written Exchange * contract_wrappers.py/doctest: rename variables * abi-gen: fix misspelling in docstring Co-Authored-By: Fabio B <[email protected]> * Py docs: error on warning, and test build in CI * abi-gen: doc Py bytes params as requiring UTF-8 * abi-gen: git mv diff.sh test-cli/ * abi-gen: put Py wrapper in module folder, not file This leaves space for user-defined additions to the same module, such as for custom types, as shown herein. * abi-gen: customizable param validation for Python * contract_wrappers.py: JSON schema Order validation * CircleCI Build Artifacts For abi-gen command-line test output, for generated Python contract wrappers as output by abi-gen, for generated Python contract wrappers as reformatted and included in the Python package area, and for the "build" output folder in each Python package, which includes the generated documentation. * CHANGELOG updates for all components * abi-gen: grammar in comments Co-Authored-By: Fabio B <[email protected]> * abi-gen: CHANGELOG spelling correction Co-Authored-By: Fabio B <[email protected]> * order_utils.py: reverse (chronological) CHANGELOG * abi-gen-templates: reset CHANGELOG patch version * CHANGELOGs: use multiple entries where appropriate * abi-gen: enable devdoc solc output in test-cli * abi-gen-templates/Py: consolidate return type * abi-gen/test-cli: non-pure fixture contract method Added a method to the "dummy" test fixture contract that isn't pure. All of the other prior method cases were pure. * abi-gen/Py: fix const methods missing return type * abi-gen/Py: fix wrong return types on some methods Specifically, wrapper methods wrapping contract methods that modify contract state and return no return value. There was no test case for this. Now there is. * contract_wrappers.py: rm generated code in `clean` * Parallelize Py monorepo scripts (test, lint, etc)
0xProject · Jul 23, 2019 · ead8099 · ead8099
1 parent 1e6e748
commit ead8099
Show file tree

Hide file tree

Showing 120 changed files with 3,245 additions and 2,116 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -33,6 +33,10 @@ jobs:
                   key: python-contract-wrappers-{{ .Environment.CIRCLE_SHA1 }}
                   paths:
                       - ~/repo/packages/python-contract-wrappers/generated
+            - store_artifacts:
+                  path: ~/repo/packages/python-contract-wrappers/generated
+            - store_artifacts:
+                  path: ~/repo/packages/abi-gen/test-cli/output
     build-website:
         resource_class: medium+
         docker:
@@ -204,7 +208,8 @@ jobs:
             - run:
                   command: |
                       cd python-packages
-                      ./cmd_pkgs_in_dep_order.py coverage run setup.py test
+                      ./parallel coverage run setup.py test
+                      ./build_docs
             - save_cache:
                   key: coverage-python-contract-addresses-{{ .Environment.CIRCLE_SHA1 }}
                   paths:
@@ -229,6 +234,24 @@ jobs:
                   key: coverage-python-sra-client-{{ .Environment.CIRCLE_SHA1 }}
                   paths:
                       - ~/repo/python-packages/sra_client/.coverage
+            - store_artifacts:
+                  path: ~/repo/python-packages/contract_wrappers/src/zero_ex/contract_wrappers/erc20_token/__init__.py
+            - store_artifacts:
+                  path: ~/repo/python-packages/contract_wrappers/src/zero_ex/contract_wrappers/exchange/__init__.py
+            - store_artifacts:
+                  path: ~/repo/python-packages/contract_addresses/build
+            - store_artifacts:
+                  path: ~/repo/python-packages/contract_artifacts/build
+            - store_artifacts:
+                  path: ~/repo/python-packages/contract_wrappers/build
+            - store_artifacts:
+                  path: ~/repo/python-packages/json_schemas/build
+            - store_artifacts:
+                  path: ~/repo/python-packages/middlewares/build
+            - store_artifacts:
+                  path: ~/repo/python-packages/order_utils/build
+            - store_artifacts:
+                  path: ~/repo/python-packages/sra_client/build
     test-rest-python:
         working_directory: ~/repo
         docker:

diff --git a/.gitignore b/.gitignore
@@ -113,6 +113,8 @@ contracts/erc1155/generated-wrappers/
 contracts/extensions/generated-wrappers/
 contracts/exchange-forwarder/generated-wrappers/
 contracts/dev-utils/generated-wrappers/
+python-packages/contract_wrappers/src/zero_ex/contract_wrappers/erc20_token/__init__.py
+python-packages/contract_wrappers/src/zero_ex/contract_wrappers/exchange/__init__.py
 
 # cli test output
 packages/abi-gen/test-cli/output
@@ -132,3 +134,6 @@ python-packages/*/dist
 __pycache__
 python-packages/*/src/*.egg-info
 python-packages/*/.coverage
+
+# python keeps package-local copies of json schemas
+python-packages/json_schemas/src/zero_ex/json_schemas/schemas
diff --git a/.prettierignore b/.prettierignore
@@ -35,3 +35,6 @@ packages/sol-coverage/test/fixtures/artifacts
 .pytest_cache
 .mypy_cache
 .tox
+packages/abi-gen/test-cli/fixtures/artifacts/AbiGenDummy.json
+packages/abi-gen/test-cli/fixtures/artifacts/LibDummy.json
+packages/abi-gen/test-cli/fixtures/artifacts/TestLibDummy.json
diff --git a/packages/0x.js/CHANGELOG.json b/packages/0x.js/CHANGELOG.json
@@ -1,4 +1,13 @@
 [
+    {
+        "version": "6.0.13",
+        "changes": [
+            {
+                "note": "re-export new ethereum-types type, TupleDataItem",
+                "pr": 1919
+            }
+        ]
+    },
     {
         "timestamp": 1563193019,
         "version": "6.0.12",

diff --git a/packages/0x.js/src/index.ts b/packages/0x.js/src/index.ts
@@ -131,6 +131,7 @@ export {
     ConstructorAbi,
     FallbackAbi,
     DataItem,
+    TupleDataItem,
     ConstructorStateMutability,
     StateMutability,
     Web3JsProvider,

diff --git a/packages/abi-gen-templates/CHANGELOG.json b/packages/abi-gen-templates/CHANGELOG.json
@@ -1,4 +1,25 @@
 [
+    {
+        "version": "2.3.0",
+        "changes": [
+            {
+                "note": "Python: fix broken event handling",
+                "pr": 1919
+            },
+            {
+                "note": "Python: custom validator class support",
+                "pr": 1919
+            },
+            {
+                "note": "Python: linter fixes",
+                "pr": 1919
+            },
+            {
+                "note": "Python: normalize bytes parameters in wrapper methods",
+                "pr": 1919
+            }
+        ]
+    },
     {
         "timestamp": 1563006338,
         "version": "2.2.1",

diff --git a/packages/abi-gen-templates/Python/contract.handlebars b/packages/abi-gen-templates/Python/contract.handlebars
@@ -1,8 +1,17 @@
 """Generated wrapper for {{contractName}} Solidity contract."""
 
+# pylint: disable=too-many-arguments
+
 import json
-from typing import Optional, Tuple, Union
+from typing import (  # pylint: disable=unused-import
+    Any,
+    List,
+    Optional,
+    Tuple,
+    Union,
+)
 
+from mypy_extensions import TypedDict  # pylint: disable=unused-import
 from hexbytes import HexBytes
 from web3.datastructures import AttributeDict
 from web3.providers.base import BaseProvider
@@ -11,13 +20,55 @@ from zero_ex.contract_wrappers._base_contract_wrapper import BaseContractWrapper
 from zero_ex.contract_wrappers.tx_params import TxParams
 
 
+class {{contractName}}ValidatorBase:
+    """Base class for validating inputs to {{contractName}} methods."""
+    def __init__(
+        self,
+        provider: BaseProvider,
+        contract_address: str,
+        private_key: str = None,
+    ):
+        """Initialize the instance."""
+
+    def assert_valid(
+        self, method_name: str, parameter_name: str, argument_value: Any
+    ):
+        """Raise an exception if method input is not valid.
+
+        :param method_name: Name of the method whose input is to be validated.
+        :param parameter_name: Name of the parameter whose input is to be
+            validated.
+        :param argument_value: Value of argument to parameter to be validated.
+        """
+
+
+# Try to import a custom validator class definition; if there isn't one,
+# declare one that we can instantiate for the default argument to the
+# constructor for {{contractName}} below.
+try:
+    # both mypy and pylint complain about what we're doing here, but this
+    # works just fine, so their messages have been disabled here.
+    from . import (  # type: ignore # pylint: disable=import-self
+        {{contractName}}Validator,
+    )
+except ImportError:
+
+    class {{contractName}}Validator({{contractName}}ValidatorBase):  # type: ignore
+        """No-op input validator."""
+
+
+{{tupleDefinitions ABIString}}
+
+
+# pylint: disable=too-many-public-methods
 class {{contractName}}(BaseContractWrapper):
-    """Wrapper class for {{contractName}} Solidity contract."""
+    """Wrapper class for {{contractName}} Solidity contract.{{docBytesIfNecessary ABIString}}"""
 
     def __init__(
         self,
         provider: BaseProvider,
         contract_address: str,
+        validator: {{contractName}}Validator = None,
         private_key: str = None,
     ):
         """Get an instance of wrapper for smart contract.
@@ -34,6 +85,11 @@ class {{contractName}}(BaseContractWrapper):
             private_key=private_key,
         )
 
+        if not validator:
+            validator = {{contractName}}Validator(provider, contract_address, private_key)
+
+        self.validator = validator
+
     def _get_contract_instance(self, token_address):
         """Get an instance of the smart contract at a specific address.
 
@@ -55,3 +111,5 @@ class {{contractName}}(BaseContractWrapper):
         return json.loads(
             '{{{ABIString}}}'  # noqa: E501 (line-too-long)
         )
+
+# pylint: disable=too-many-lines
diff --git a/packages/abi-gen-templates/Python/partials/call.handlebars b/packages/abi-gen-templates/Python/partials/call.handlebars
@@ -6,31 +6,44 @@
         {{^this.constant}}
         view_only: bool = False,
         {{/this.constant}}
-    ) ->{{#if outputs}}{{~> return_type outputs=outputs~}}{{else}} None{{/if}}:
+    ) -> {{> return_type outputs=outputs~}}:
         """Execute underlying, same-named contract method.
-{{sanitizeDevdocDetails this.name this.devdoc.details 8}}
-        {{#each this.devdoc.params}}
-        :param {{@key}}: {{this}}
-        {{/each}}
-        {{#if this.constant}}
-        {{#if this.devdoc.return}}:returns: {{this.devdoc.return}}{{/if}}
-        {{else}}
+{{sanitizeDevdocDetails this.name this.devdoc.details 8}}{{~#if this.devdoc.params~}}{{#each this.devdoc.params}}
+{{makeParameterDocstringRole @key this 8}}{{/each}}{{/if}}
         :param tx_params: transaction parameters
+        {{#if this.constant~}}
+        {{#if this.devdoc.return}}
+{{makeReturnDocstringRole this.devdoc.return 8}}{{/if}}
+        {{else}}
         :param view_only: whether to use transact() or call()
 
-        :returns: transaction hash
+        :returns: if param `view_only`:code: is `True`:code:, then returns the
+            value returned from the underlying function; else returns the
+            transaction hash.
         {{/if}}
         """
         {{#each this.inputs}}
+        self.validator.assert_valid(
+            method_name='{{../name}}',
+            parameter_name='{{name}}',
+            argument_value={{toPythonIdentifier name}},
+        )
         {{#if (equal type 'address')}}
-        {{this.name}} = self._validate_and_checksum_address({{this.name}})
+        {{toPythonIdentifier this.name}} = self._validate_and_checksum_address({{toPythonIdentifier this.name}})
         {{else if (equal type 'uint256')}}
         # safeguard against fractional inputs
-        {{this.name}} = int({{this.name}})
+        {{toPythonIdentifier this.name}} = int({{toPythonIdentifier this.name}})
+        {{else if (equal type 'bytes')}}
+        {{toPythonIdentifier this.name}} = bytes.fromhex({{toPythonIdentifier this.name}}.decode("utf-8"))
+        {{else if (equal type 'bytes[]')}}
+        {{toPythonIdentifier this.name}} = [
+            bytes.fromhex({{toPythonIdentifier this.name}}_element.decode("utf-8"))
+            for {{toPythonIdentifier this.name}}_element in {{toPythonIdentifier this.name}}
+        ]
 {{/if}}
 {{/each}}
         func = self._get_contract_instance(
-            self._contract_address
+            self.contract_address
         ).functions.{{this.name}}(
             {{> params}}
         )

diff --git a/packages/abi-gen-templates/Python/partials/event.handlebars b/packages/abi-gen-templates/Python/partials/event.handlebars
@@ -1,14 +1,13 @@
     def get_{{languageSpecificName}}_event(
-        self, token_address: str, tx_hash: Union[HexBytes, bytes]
+        self, tx_hash: Union[HexBytes, bytes]
     ) -> Tuple[AttributeDict]:
         """Get log entry for {{name}} event.
 
-        :param tx_hash: hash of transaction emitting {{name}} event.
+{{makeEventParameterDocstringRole name 8}}
         """
         tx_receipt = self._web3_eth.getTransactionReceipt(tx_hash)
-        token_address = self._validate_and_checksum_address(token_address)
         return (
-            self._get_contract_instance(token_address)
+            self._get_contract_instance(self.contract_address)
             .events.{{name}}()
             .processReceipt(tx_receipt)
         )
diff --git a/packages/abi-gen-templates/Python/partials/params.handlebars b/packages/abi-gen-templates/Python/partials/params.handlebars
@@ -1,3 +1,3 @@
 {{#each inputs}}
-{{name}}{{#if @last}}{{else}},{{/if}}
+{{toPythonIdentifier name}}{{#if @last}}{{else}},{{/if}}
 {{/each}}
diff --git a/packages/abi-gen-templates/Python/partials/return_type.handlebars b/packages/abi-gen-templates/Python/partials/return_type.handlebars
@@ -1,13 +1,14 @@
-{{#if this.constant}}
+{{~#if outputs~}}
+{{^if this.constant}}
+Union[
+{{~/if~}}
 {{#if outputs.length}}
 {{#singleReturnValue}}
- {{#returnType outputs.0.type outputs.0.components}}{{~/returnType~}}
+{{#returnType outputs.0.type outputs.0.components}}{{~/returnType~}}
 {{/singleReturnValue}}
 {{^singleReturnValue}}
 [{{#each outputs}}{{#returnType type components}}{{/returnType}}{{#unless @last}}, {{/unless}}{{/each}}]
 {{/singleReturnValue}}
-{{/if}}
-{{/if}}
-{{^if this.constant}}
- Union[HexBytes, bytes]
-{{~/if~}}
+{{else}}None
+{{/if}}{{^if this.constant}}, Union[HexBytes, bytes]]{{/if~}}
+{{else}}{{#if this.constant}}None{{else}}Union[None, Union[HexBytes, bytes]]{{/if}}{{/if~}}
diff --git a/packages/abi-gen-templates/Python/partials/typed_params.handlebars b/packages/abi-gen-templates/Python/partials/typed_params.handlebars
@@ -1,3 +1,3 @@
 {{#each inputs}}
-{{name}}: {{#parameterType type components}}{{/parameterType}},
+{{toPythonIdentifier name}}: {{#parameterType type components}}{{/parameterType}},
 {{/each}}
diff --git a/packages/abi-gen/CHANGELOG.json b/packages/abi-gen/CHANGELOG.json
@@ -1,4 +1,37 @@
 [
+    {
+        "version": "3.1.1",
+        "changes": [
+            {
+                "note": "Python method parameters are now in snake case",
+                "pr": 1919
+            },
+            {
+                "note": "Python wrappers now support tuples in method parameters",
+                "pr": 1919
+            },
+            {
+                "note": "document Python method's bytes params as requiring UTF-8",
+                "pr": 1919
+            },
+            {
+                "note": "generate Python output into a contract-named folder, not a file (eg exchange/__init__.py rather than exchange.py) leaving space for user-defined additions to the same module, such as for custom types, as used by the Exchange wrapper's manually-written type aliases in the contract_wrappers.exchange.types Python module",
+                "pr": 1919
+            },
+            {
+                "note": "support for customizable parameter validation for Python wrappers",
+                "pr": 1919
+            },
+            {
+                "note": "wrap Python docstrings better, for pydocstyle compliance",
+                "pr": 1919
+            },
+            {
+                "note": "lots of fixes to satisfy linters of generated Python code",
+                "pr": 1919
+            }
+        ]
+    },
     {
         "timestamp": 1563047529,
         "version": "2.1.1",

diff --git a/packages/abi-gen/compiler.json b/packages/abi-gen/compiler.json
@@ -17,7 +17,8 @@
                     "evm.bytecode.object",
                     "evm.bytecode.sourceMap",
                     "evm.deployedBytecode.object",
-                    "evm.deployedBytecode.sourceMap"
+                    "evm.deployedBytecode.sourceMap",
+                    "devdoc"
                 ]
             }
         }

diff --git a/packages/abi-gen/package.json b/packages/abi-gen/package.json
@@ -33,7 +33,7 @@
         "generate_contract_wrappers": "run-p gen_typescript gen_python",
         "gen_typescript": "abi-gen --abis  ${npm_package_config_abis} --template ../../node_modules/@0x/abi-gen-templates/contract.handlebars --partials '../../node_modules/@0x/abi-gen-templates/partials/**/*.handlebars' --output ./test-cli/output/typescript --backend ethers",
         "gen_python": "abi-gen --abis  ${npm_package_config_abis} --template ../../node_modules/@0x/abi-gen-templates/Python/contract.handlebars --partials '../../node_modules/@0x/abi-gen-templates/Python/partials/**/*.handlebars' --output ./test-cli/output/python --language Python",
-        "diff_contract_wrappers": "./diff.sh ./test-cli/expected-output ./test-cli/output",
+        "diff_contract_wrappers": "test-cli/diff.sh ./test-cli/expected-output ./test-cli/output",
         "coverage:report:text": "istanbul report text",
         "coverage:report:html": "istanbul report html && open coverage/index.html",
         "profiler:report:html": "istanbul report html && open coverage/index.html",