compile/migrate with multiple solidity versions crashes ganache 2

[BlinkyStitt]

• I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

I'm writing a contract with solidity 0.5 and wanting to call a contract written (and deployed by someone else on mainnet) with solidity 0.4 (and another with Vyper: #1623). How should I setup truffle+ganache in development to deploy the old contract with the old compiler and then deploy my new contracts? I don't want to update their contract to 0.5 because I want my tests to use the version they already have deployed on mainnet. I'm using an interface to interact with the old code as detailed at https://solidity.readthedocs.io/en/v0.5.0/050-breaking-changes.html#interoperability-with-older-contracts

The 0.4 contract (https://github.com/projectchicago/gastoken/blob/master/contract/GST2_ETH.sol) is using python tools for deployment. If it were using truffle, I could run truffle migrate from their directory and then run truffle migrate from mine, but then I'm not sure how I would use artifacts.

If I symlink their *.sols into my contracts directory, it fails with the expected "ParserError: Source file requires different compiler version"

Right now I'm just hard coding the gastoken contract address so I can test the other parts of my migrations, but that won't work once I actually want to test using the gastoken contract from inside my own contract.

Is there some way to deploy their .asm files directly? I feel like this should be a common need, but I'm not seeing any documentation about it.

And when I do get to putting my contracts on mainnet, what is the best practice for linking to the already deployed gastoken contract instead of deploying it myself?

[gnidan]

Thanks for raising this concern! It's on the roadmap to support multiple solc versions natively, but there are some technical limitations that make this not immediately feasible.

The component you are missing is that you need proper Truffle artifact files (build/contracts/ JSON files) for each contract. You can't deploy ASM directly, but if you get the bytecode, you can wrap the bytecode in an artifact file and deploy that.

You might be able to achieve a workaround by way of using Truffle's new external compiler configuration. It should be flexible enough for this use case, but let me know.

[BlinkyStitt]

Thanks!

I added https://github.com/projectchicago/gastoken as a git submodule at ./external/gastoken. That repo includes *.sol, *.abi, and *.asm files that I think are relevant.

Then I added this to my truffle-config.json:

module.exports = {
    ...
    external: {
      command: "true",
      targets: [{
        properties: {
          contractName: "GST2",
        },
        fileProperties: {
          abi: "./external/gastoken/contract/GST.abi",
          bytecode: "./external/gastoken/contract/GST2_ETH.asm",
        },
      }]
    }
  },

And now truffle compile places a GST2.json file into my contracts directory. So I think this is good progress, but it isn't right yet (deploying runs out of gas). You said "you can't deploy ASM directly, but if you get the bytecode, you can wrap the bytecode." How do I do that? Do I need to set command to "solc --something"? I was hoping that I could use the .abi and .asm files that came with the git clone rather than compiling.

Starting migrations...
======================
> Network name:    'development'
> Network id:      1546548964821
> Block gas limit: 6721975


2_deploy_contracts.js
=====================

   Deploying 'GST2'
   ----------------
Error:  *** Deployment Failed ***

"GST2" ran out of gas. Something in the constructor (ex: infinite loop) caused gas estimation to fail. Try:
   * Making your contract constructor more efficient
   * Setting the gas manually in your config or as a deployment parameter
   * Using the solc optimizer settings in 'truffle.js'
   * Setting a higher network block limit if you are on a
     private network or test client (like ganache).

    at /contracts/node_modules/truffle/build/webpack:/packages/truffle-deployer/src/deployment.js:364:1
    at process._tickCallback (internal/process/next_tick.js:68:7)
Truffle v5.0.1 (core: 5.0.1)
Node v10.14.2
error Command failed with exit code 1.

I tried compiling myself with solidity 0.4.10, but I get the same deployment failed message:

    external: {
      command: "./old_solidity/node_modules/.bin/solcjs -o build --abi --bin external/gastoken/contract/GST2_ETH.sol external/gastoken/contract/rlp.sol ",
      targets: [{
        properties: {
          contractName: "GST2",
        },
        fileProperties: {
          abi: "build/external_gastoken_contract_GST2_ETH_sol_GasToken2.abi",
          bytecode: "build/external_gastoken_contract_GST2_ETH_sol_GasToken2.bin",
        },
      }]
    }
  },

Increasing the gas limit doesn't help.

   Deploying 'GST2'
   ----------------
Error:  *** Deployment Failed ***

"GST2" ran out of gas (using a value you set in your network config or deployment parameters.)
   * Block limit:  6721975
   * Gas sent:     6000000

    at /contracts/node_modules/truffle/build/webpack:/packages/truffle-deployer/src/deployment.js:364:1
    at process._tickCallback (internal/process/next_tick.js:68:7)
Truffle v5.0.1 (core: 5.0.1)
Node v10.14.2

Is "bytecode" different than what I get with --bin? This contract deployed successfully on mainnet with 1396556 gas limit: https://etherscan.io/tx/0x5de8a3c310eb8e1bd90fcbf1bde073efc2ed3750e60f207d75a993ac40a0dda6

[BlinkyStitt]

It appears that bytecode is different than what solc --bin gives. EDIT: Nope. This was only because I had the optimzer enabled with --standard-json, but had forgotten to turn it on when I was calling solc.

solc --help does not show any flags for outputting bytecode, so I'm guessing that --bin is what I want.

I've got something working, but I'm not sure how correct it is.

I ended up making an input json which does allow the necessary output selection options:

{
  "language": "Solidity",
  "sources":
  {
    "GST2_ETH.sol":
    {
      "urls":
      [
        "./GST2_ETH.sol"
      ]
    }
  },
  "settings":
  {
    "optimizer": {
      "enabled": true,
      "runs": 200
    },
    "outputSelection": {
      "*": {
        "*": [ "*" ]
      }
    }
  }
}

Then I made a compile script:

#!/bin/bash -eu

echo "Building external contracts..."

script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

root_dir="$(realpath "${script_dir}/../")"

build_dir="${root_dir}/build"

external_dir="${root_dir}/external"

mkdir -p "$build_dir/tmp"

# pull bytecode and abi out of solc --standard-json output
function extract_abi_and_bytecode {
    build_json=$1
    output_dir=$2
    sol_filename=$3
    contract_name=$4

    jq ".contracts[\"$sol_filename\"][\"$contract_name\"].abi" \
        < "$build_json" \
        > "${output_dir}/${contract_name}.abi"

    echo "Created \"${output_dir}/${contract_name}.abi\""

    # prefix bytecode with 0x
    jq -j "\"0x\"+.contracts[\"$sol_filename\"][\"$contract_name\"].evm.bytecode.object" \
        < "$build_json" \
        > "${output_dir}/${contract_name}.bytecode"

    echo "Created \"${output_dir}/${contract_name}.bytecode\""
}

# gastoken
cd "$external_dir/gastoken/contract"

cat "$external_dir/solc-gastoken.json"
solc-0.4.25 \
    --allow-paths "$(pwd)" \
    --standard-json \
    < "$external_dir/solc-gastoken.json" \
    > "$build_dir/tmp/gastoken.json"

extract_abi_and_bytecode "$build_dir/tmp/gastoken.json" "$build_dir/gastoken" "GST2_ETH.sol" "GasToken2"

echo "Success building external contracts"

And then I added this script to my truffle-config.js

module.exports = {
  compilers: {
    // TODO: only run if they are outdated
    ...
    external: {
      command: "./scripts/build-external.sh",
      targets: [
        {
          properties: {
            contractName: "GasToken2",
          },
          fileProperties: {
            abi: "build/gastoken/GasToken2.abi",
            bytecode: "build/gastoken/GasToken2.bytecode",
          },
        },
        // TODO: lots more contracts here
      ],
    },
  },
  ...
};

And now my migrations work!

I think this would be a lot simpler if truffle could use the output of --bin.

[BlinkyStitt]

I was able to simplify my external compile script:

#!/bin/bash -eux

echo "Building external contracts..."

script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

root_dir="$(realpath "${script_dir}/../")"

build_dir="${root_dir}/build"

external_dir="${root_dir}/external"

# gastoken
gastoken_contract_dir="${external_dir}/gastoken/contract"
solc-0.4.25 \
    --allow-paths "${gastoken_contract_dir}" \
    --optimize \
    --output-dir "${build_dir}/gastoken" \
    --overwrite \
    --abi --ast-json --bin \
    "=${gastoken_contract_dir}/" \
    "${gastoken_contract_dir}/GST2_ETH.sol"

# bin output needs to be prefixed with 0x for truffle to like it # TODO: do this for ALL $build_dir/*/*.bin
(echo -n "0x"; cat "${build_dir}/gastoken/GasToken2.bin") > "${build_dir}/gastoken/GasToken2.bytecode"

echo "Success building external contracts"

This would be simpler if truffle didn't require that bytecode started with "0x". Why does it require that? solc doesn't output it that way, so I'm having to modify its output.

[BlinkyStitt]

Where are the available "fileProperties" documented? I needed to add ast to it to get slither working. I don't think the way I added it is working with ganache 2.0.0-beta.2 though.

truffle-config.js:

    external: {
      command: "./scripts/build-external.sh",
      targets: [
        {
          properties: {
            contractName: "GasToken2",
          },
          fileProperties: {
            abi: "build/external/gastoken/GasToken2.abi",
            ast: "build/external/gastoken/GST2_ETH.sol_json.ast",
            bytecode: "build/external/gastoken/GasToken2.truffle-bin",
          },
        },
        {
          properties: {
            contractName: "rlp",
          },
          fileProperties: {
            abi: "build/external/gastoken/Rlp.abi",
            ast: "build/external/gastoken/rlp.sol_json.ast",
            bytecode: "build/external/gastoken/Rlp.truffle-bin",
          },
        },

trufle compile/migrate then outputs this GasToken2.json: https://gist.github.com/WyseNynja/e54064191465abc3c662b3b2df1e4280

While truffle test works, when I look at the contract in Ganache it tries to load for a few seconds and then crashes with this error:

TypeError: Cannot read property 'ast' of undefined
    at Object.getContractNode (/node_modules/truffle-decoder/dist/interface/contract-decoder.js:37:33)
    at getStateVariables (/node_modules/truffle-decoder/dist/allocate/references.js:187:45)
    at Object.getContractStateVariables (/node_modules/truffle-decoder/dist/allocate/references.js:209:27)
    at TruffleContractDecoder.<anonymous> (/node_modules/truffle-decoder/dist/interface/contract-decoder.js:78:55)
    at Generator.next (<anonymous>)
    at /node_modules/truffle-decoder/dist/interface/contract-decoder.js:7:71
    at Promise (<anonymous>)
    at __awaiter (/node_modules/truffle-decoder/dist/interface/contract-decoder.js:3:12)
    at TruffleContractDecoder.init (/node_modules/truffle-decoder/dist/interface/contract-decoder.js:69:16)
    at Object.getContractState (/src/truffle-integration/decode.js:9:11)
    at process.<anonymous> (/src/truffle-integration/index.js:78:52)
    at emitTwo (events.js:125:13)
    at process.emit (events.js:213:7)
    at emit (internal/child_process.js:768:12)
    at _combinedTickCallback (internal/process/next_tick.js:141:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[BlinkyStitt]

I'm thinking of using https://sol-compiler.com/ instead of manually calling solc. They have support for compiling contracts with different versions of solidity. I don't see any clear way to convert from solc's combined json output to a truffle artifact though. Is there any existing tooling for that? I'd like to be able to keep using truffle migrate for managing all the contracts even if truffle isn't the one compiling them.

[eggplantzzz]

@wysenynja Let me see if I can locate some helpful material for you. From what I understand you are trying to build a truffle contract from scratch and want guidance, sound about right? Sorry for the delay in responding to you by the way.

[eggplantzzz]

I think the schema for those artifacts can be found here --> https://github.com/trufflesuite/truffle/tree/develop/packages/truffle-contract-schema

Also require("truffle-contract-schema").validate(obj) can help you validate the contract object.

[BlinkyStitt]

@eggplantzzz Thanks for the reply. I'm trying to let truffle build the truffle contract during the external compile step. I'm wanting to know the best way to convert solc's output to a truffle contract. I can see a couple ways of doing it. I thought using a truffle-config.js like this would put everything in the right place, but it only seems to setup the abi and bytecode:

    external: {
      command: "./scripts/build-external.sh",
      targets: [
        {
          properties: {
            contractName: "GasToken2",
            sourcePath: "external/gastoken/contract/GST2_ETH.sol",
          },
          fileProperties: {
            abi: "build/external/gastoken/GasToken2.abi",
            ast: "build/external/gastoken/GST2_ETH.sol_json.ast",
            bytecode: "build/external/gastoken/GasToken2.truffle-bin",
            deployedBytecode: "build/external/gastoken/GasToken2.truffle-bin-runtime",
            devdoc: "build/external/gastoken/GasToken2.docdev",
            userdoc: "build/external/gastoken/GasToken2.docuser",
            source: "external/gastoken/contract/GST2_ETH.sol",
          },
        },

I would much prefer to be able to use the combined json from solc instead of having to configure all those properties manually for each contract. To do that, it sounds like I need a script that reads solc's combined output json and builds something that require("truffle-contract-schema").validate(obj) is happy with. This script would place the artifacts itself instead of having targets in truffle-config.

Rather than an external script, I think it would be better for this code (both supporting more properties and reading solc's combined json) to be a part of the truffle external compiler step. Where in truffle's code is the abi and bytecode passed through? It seems like adding support for source maps and the others should be relatively simple. Loading the combined json would be more of a change, but it would certainly be helpful.

[gnidan]

@wysenynja just a drive-by response: you might be interested to look at the integration tests for this feature. They use truffle compile itself as an external compiler.

Right now, the translation between Solidity standard JSON and a Truffle artifact is done by truffle-compile, and it relies on Truffle tightly controlling the input. It's not entirely straightforward to take any solc output and convert it to artifacts, at least not enough to provide out of the box.

I would recommend maybe looking at the path version of targets and maybe building a truffle run plugin? Something like:

module.exports = {
  /* ... */
  compilers: {
    external: {
      command: "...",
      targets: [{
        path: ".../path/to/solc-outputs/*.solc.json"
        command: "truffle run solc-standard-json-to-artifact"
      }]
    }
  }
}

[eggplantzzz]

https://github.com/trufflesuite/truffle/blob/develop/packages/truffle-compile/index.js#L194
This is the line where the code puts together the contract.

[BlinkyStitt]

I'm trying out @gnidan's suggestion and I think I am getting closer to what I need.

Here is my truffle-config.js's external section:

    external: {
      command: "sol-compiler",
      targets: [
        {
          path: "build/external/sol-compiler/*.json",
          command: "node sol-compiler-json-to-truffle.js",
          stdin: false,
        },
      ],
    },

Here is what I've come up with for an external compile target script so far:

#!/usr/bin/env node
const fs = require('fs');
const semver = require('semver');

// node sol-compiler-json-to-truffle.js inputFilename
const inputFilename = process.argv[2];

const inputObj = JSON.parse(fs.readFileSync(inputFilename, 'utf8'));

if (!semver.satisfies(inputObj["schemaVersion"], '^2.0.0')) {
  throw "incompatible schemaVersion";
}

var truffleObject = {
  "contractName": inputObj["contractName"],
  "abi": inputObj["compilerOutput"]["abi"],
  "bytecode": inputObj["compilerOutput"]["evm"]["bytecode"]["object"],
  "deployedBytecode": inputObj["compilerOutput"]["evm"]["deployedBytecode"]["object"],
  "sourceMap": inputObj["compilerOutput"]["evm"]["bytecode"]["sourceMap"],
  "deployedSourceMap": inputObj["compilerOutput"]["evm"]["deployedBytecode"]["sourceMap"],
  // TODO: "source": inputObj["sources"][inputObj["contractName"] + ".sol"],
  // TODO: "sourcePath": "",
  // TODO: "ast": {},
  // TODO: "lagacyAST": {},
  "compiler": {
    "name": inputObj["compiler"]["name"],
    "version": inputObj["compiler"]["version"],
  },
  "networks": {},
  "schemaVersion": "3.0.1",
  "updatedAt": "2019-01-27T03:25:52.161Z",  // TODO: dynamic
  "devdoc": inputObj["compilerOutput"]["devdoc"],
  "userdoc": inputObj["compilerOutput"]["userdoc"],
};

// TODO: check somewhere for mainnet contract addresses

// TODO: validate truffleObject here? I think truffle will do it for us after this

// console.error(truffleObject);

// give truffle the contract artifact formatted like it expects
console.log(JSON.stringify(truffleObject));

Something is wrong about the abi though. Running truffle compile gives this error along with a bunch of others:

Error: Schema validation failed. Errors:

should NOT have additional properties (additionalProperties):
{ dataPath: '.abi[3]',
  schemaPath: '#/additionalProperties',
  params: { additionalProperty: 'constant' },
  data:
   { constant: true,
     inputs:
      [ { name: '_addresses', type: 'address[]' },
        { name: '_calldatas', type: 'bytes[]' } ],
     name: 'encodeContractCalls',
     outputs: [ { name: 'script', type: 'bytes' } ],
     payable: false,
     stateMutability: 'pure',
     type: 'function' },
  parentSchema:
   { type: 'object',
     properties:
      { type: { type: 'string', enum: [ 'event' ] },
        name: { '$ref': '#/definitions/Name' },
        inputs:
         { type: 'array',
           items: { '$ref': '#/definitions/EventParameter' } },
        anonymous: { type: 'boolean' } },
     required: [ 'type', 'name', 'inputs', 'anonymous' ],
     additionalProperties: false } }

I don't understand why it thinks that "constant" is an additional property. Looking at the truffle artifact that truffle generated itself, the abi sections look pretty much the same to me. The order is different between them, but all the keys are the same. What sort of modifications do I need to make to abi? Is sharing my complete repo needed to debug this?

If I set abi to [] it gets further, so I think this is the right track.

[BlinkyStitt]

Here is the full output I get when I try to include the abi as sol-compiler gives it to me: https://gist.github.com/WyseNynja/d1536bf19ff754c2bbc0f3d6aa0508e2

The good news is that I think fixing why the abi won't validate is the last step. If I set abi to [] I am able to migrate the contracts with truffle migrate.

Thanks for the help so far, everyone!

[BlinkyStitt]

The problem with https://gist.github.com/WyseNynja/d1536bf19ff754c2bbc0f3d6aa0508e2 was using pragma experimental ABIEncoderV2; and bytes[]

After commenting out the method that was doing that, the abi validates. I guess I can work without that for now, but it was nice to have.

[BlinkyStitt]

Given that the rest of my issues seem to be in ganache, I'm going to continue this at trufflesuite/ganache#1117