Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Truffle3, refactor CLI, add CLI unit tests, fix misc bugs #1

Merged
merged 54 commits into from
May 1, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
b1fa110
Add truffle fork dep, exec.js
cgewecke Mar 28, 2017
f1de0f4
exec.js test suite updated for truffle 3
cgewecke Mar 29, 2017
31125bf
Toggle silence off to debug CI failures
cgewecke Mar 29, 2017
bb50f66
Comment out suite flush test, remove truffle -g from circle.yml
cgewecke Mar 29, 2017
fa903ec
Comments/cleanup in exec.js. Add --test flag, add CI logic to flush test
cgewecke Mar 29, 2017
95f0580
Remove .only on run tests
cgewecke Mar 29, 2017
d9d9789
Attempt to normalize paths for "npm install --save-dev" use
cgewecke Mar 30, 2017
b24dbc0
Remove patching - replace with testrpc/ethereumjs-vm forks
cgewecke Mar 30, 2017
6de8846
Remove silence to debug CI failure, lengthen timeout, npm bin
cgewecke Mar 30, 2017
6fb6f0b
Add debugging line for norpc, explicitly include yargs
cgewecke Mar 30, 2017
adbdf8e
Try removing shebang to get CI to pass
cgewecke Mar 30, 2017
60c9332
Try to fix CI: remove yargs, add commander for argv processing
cgewecke Mar 30, 2017
9f770c8
Move to config file based options, allow use of coverage network in t…
cgewecke Mar 31, 2017
c98c53b
Debug CI - untoggle silent
cgewecke Mar 31, 2017
1bf4ac3
Only run single testrpc instance in tests
cgewecke Mar 31, 2017
e075842
Use istanbul API instead of shell.exec b/c incompatible w/ npm insta…
cgewecke Mar 31, 2017
f6b617e
Debug CI: run flush test
cgewecke Mar 31, 2017
5228fe7
Exit process at end (travis got stuck if not)
Apr 12, 2017
da24a90
Allows contract inheritance
Apr 12, 2017
128dd31
Added keccakjs dependency
Apr 12, 2017
fbfd049
eslint-ed
Apr 12, 2017
2004589
Updated solc to 0.4.10
Apr 12, 2017
7012cda
reverted version to solc 0.4.8
Apr 12, 2017
9d24ac7
Remove some extraneous deps, use sc-forks testrpc
cgewecke Apr 17, 2017
440e685
Use sc-forks, add inheritance test, fix report bug
cgewecke Apr 18, 2017
108d7a0
Try resolving merge conflict
cgewecke Apr 19, 2017
4cc0ad5
Remove runCoveredTests.js
cgewecke Apr 19, 2017
cb833aa
Merge branch 'adriamb-master' into truffle3-adriamb
cgewecke Apr 19, 2017
d8eafb8
Merge adriamb inheritance fix, use non-forked (global) truffle
cgewecke Apr 19, 2017
c4f03dc
Lint
cgewecke Apr 19, 2017
af94780
Lint II
cgewecke Apr 19, 2017
1bea757
Lint III: add .eslintignore for truffle-crash test (intentional synta…
cgewecke Apr 19, 2017
3fd5e32
Merge pull request #58 from cgewecke/truffle3-adriamb
cgewecke Apr 19, 2017
c98d085
Fix testrpc fork name in pkg, remove unused testing utils
cgewecke Apr 19, 2017
a46f733
Accomodate both locations NPM might install testrpc-sc
cgewecke Apr 19, 2017
52f0a4b
Add assembly AST node names, edit README for npm install
cgewecke Apr 19, 2017
7d652aa
Generate a config for truffle.js networks -> development, instead of …
cgewecke Apr 19, 2017
9af5c30
Revert previous config fix, update README to require a development ne…
cgewecke Apr 19, 2017
3e53edc
Add missing assembly AST node type, ignore client truffle.js, revert …
cgewecke Apr 20, 2017
0b57ebc
Accomodate LibraryStatements in instrumentSolidity.js
cgewecke Apr 21, 2017
695b2fe
Get truffle exec to recognize networks option flag
cgewecke Apr 21, 2017
818689d
Update readme for networks option
cgewecke Apr 21, 2017
eacbe63
Run testrpc with 35 accounts (uport-proxy) / allow configuration
cgewecke Apr 21, 2017
440bd8b
Fix chained constructor/method call bug. Add compilation unit test fo…
cgewecke Apr 23, 2017
769c7f8
Add lint script to package.json
cgewecke Apr 23, 2017
21d3c04
Fix empty contract body parsing bug. Add coverage unit test for case
cgewecke Apr 24, 2017
e65a035
Update README with known issues, links to zeppelin example report
cgewecke Apr 24, 2017
381285c
Allow testrpc options string in config, rename run to cli (test seque…
cgewecke Apr 24, 2017
7617134
Add unit test for arbitrary testing command config option, remove tes…
cgewecke Apr 24, 2017
4b627f6
Fix broken chained call handling, add unit tests to verify cases get …
cgewecke Apr 24, 2017
8223aec
Rename "run" folders/files "cli" for consistency
cgewecke Apr 24, 2017
2f933a6
Disable two "config" tests for CI - multiple testrpc launches max out…
cgewecke Apr 24, 2017
c47f160
README updates for .solcover.js config options, more example reports
cgewecke Apr 25, 2017
48e77b2
Misc edits for clarity
cgewecke Apr 25, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
test/cli/truffle-crash.js
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@ allFiredEvents
coverage.json
coverage/
node_modules/
.changelog
.DS_Store
127 changes: 85 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,75 +1,118 @@
#SolCover
# SolCover

![CircleCI Status](https://circleci.com/gh/JoinColony/solcover.svg?style=shield&circle-token=53d5360d290ef593c7bdce505b86ae8b9414e684)
[![codecov](https://codecov.io/gh/JoinColony/solcover/branch/master/graph/badge.svg)](https://codecov.io/gh/JoinColony/solcover)


###Code coverage for Solidity testing
### Code coverage for Solidity testing
![coverage example](https://cdn-images-1.medium.com/max/800/1*uum8t-31bUaa6dTRVVhj6w.png)

For more details about what this is, how it work and potential limitations, see
For more details about what this is, how it works and potential limitations, see
[the accompanying article](https://blog.colony.io/code-coverage-for-solidity-eecfa88668c2).

###Installation and preparation
### Install
```
$ npm install --save-dev https://github.com/JoinColony/solcover.git#truffle3
```

From your truffle directory, clone this repo:
### Run
```
git clone http://github.com/JoinColony/solcover.git
cd solcover
npm install
$ ./node_modules/solcover/exec.js
```

Until [Truffle allows the `--network` flag for the `test` command](https://github.com/ConsenSys/truffle/issues/239), in `truffle.js` you have to set a large gas amount for deployment. While this is set, uninstrumented tests likely won't run correctly, so this should only be set when running the coverage tests. An appropriately modified `truffle.js` might look like
Tests run signficantly slower while coverage is being generated. A 1 to 2 minute delay
between the end of Truffle compilation and the beginning of test execution is possible if your
test suite is large. Large solidity files can also take a while to instrument.

```
### Configuration

By default, Solcover generates a stub `truffle.js` that accomodates its special gas needs and
connects to a modified version of testrpc on port 8555. If your tests will run on the development network
using a standard `truffle.js` and a testrpc instance with no special options, you shouldn't have to
do any configuration. If your tests depend on logic added to `truffle.js` - for example:
[zeppelin-solidity](https://github.com/OpenZeppelin/zeppelin-solidity/blob/master/truffle.js)
uses the file to expose a babel polyfill that its suite requires - you can override the
default behavior by declaring a coverage network in `truffle.js`. Solcover will use your 'truffle.js'
instead of a dynamically generated one.

**Example coverage network config**
```javascript
module.exports = {
rpc: {
host: 'localhost',
gasPrice: 20e9,
gas: 0xfffffff,
networks: {
development: {
host: "localhost",
port: 8545,
network_id: "*" // Match any network id
},
coverage: {
host: "localhost",
network_id: "*",
port: 8555, // <-- Use port 8555
gas: 0xfffffffffff, // <-- Use this high gas value
gasPrice: 0x01 // <-- Use this low gas price
}
}
};
};
```
In the future, hopefully just adding the 'coverage' network to `truffle.js` will be enough. This will look like

You can also create a `.solcover.js` config file in the root directory of your project and specify
some additional options:
+ **port**: <Number> The port you want Solcover to run testrpc on / have truffle connect to.
+ **testrpcOptions**: <String> A string of options to be appended to a command line invocation
of testrpc.
+ Example: `--account="0x89a...b1f',10000" --port 8777`".
+ Note: you should specify the port in your `testrpcOptions` string AND as a `port` option.
+ **testCommand**: <String> By default Solcover runs `truffle test` or `truffle test --network coverage`.
This option lets you run tests some other way: ex: `mocha --timeout 5000`. You
will probably also need to make sure the web3 provider for your tests explicitly connects to the port Solcover's
testrpc is set to run on, e.g:
+ `var web3 = new Web3(new Web3.providers.HttpProvider("http://localhost:8555"))`

**Example .solcover.js config file**
```
module.exports = {
rpc: {
host: 'localhost',
gasPrice: 20e9,
},
networks:{
"coverage":{
gas: 0xfffffff,
}
}
}
port: 6545,
testrpcOptions: '-p 6545 -u 0x54fd80d6ae7584d8e9a19fe1df43f04e5282cc43',
testCommand: 'mocha --timeout 5000'
};
```
and will not interfere with normal `truffle test` - or other commands - being run during development.

Note that if you have hardcoded gas costs into your tests, some of them may fail when using SolCover. This is because the instrumentation process increases the gas costs for using the contracts, due to the extra events. If this is the case, then the coverage may be incomplete. To avoid this, using `estimateGas` to estimate your gas costs should be more resilient in most cases.

###Execution

Firstly, make sure that your contracts in your truffle directory are saved elsewhere too - this script moves them and modifies them to do the instrumentation and allow `truffle` to run the tests with the instrumented contracts. It returns them after the tests are complete, but if something goes wrong, then `originalContracts` in the truffle directory should contain the unmodified contracts.
### Known Issues

SolCover runs its own (modified) `testrpc` to get the coverage data, so make sure that you've not left a previous instance running on port 8545, otherwise the coverage reported will be.... sparse...
**Hardcoded gas costs**: If you have hardcoded gas costs into your tests some of them may fail when using SolCover.
This is because the instrumentation process increases the gas costs for using the contracts, due to
the extra events. If this is the case, then the coverage may be incomplete. To avoid this, using
`estimateGas` to estimate your gas costs should be more resilient in most cases.

From inside the SolCover directory, run
**Events testing**: Because Solcover injects events into your contracts to log which lines your tests reach,
any tests that ask how many events are fired or where the event sits in the logs array
will probably error while coverage is being generated.

```node ./runCoveredTests.js```
**Using `require` in `migrations.js` files**: Truffle overloads Node's `require` function but
implements a simplified search algorithm for node_modules packages
([see issue #383 at Truffle](https://github.com/trufflesuite/truffle/issues/383)).
Because Solcover copies an instrumented version of your project into a temporary folder, `require`
statements handled by Truffle internally won't resolve correctly.

Upon completion of the tests, open the `./coverage/lcov-report/index.html` file to browse the HTML coverage report.
**Coveralls / CodeCov**: These CI services take the Istanbul reports generated by Solcover and display
line coverage. Istanbul's own html report publishes significantly more detail and can show whether
your tests actually reach all the conditional branches in your code. It can be found inside the
`coverage` folder at `index.html` after you run the tool.

###A few, uh, provisos, a, a couple of quid pro quos...
It is very likely that there are valid Solidity statements that this tool won't instrument correctly, as it's only been developed against a small number of contracts. If (and when) you find such cases, please raise an issue.
### Examples

+ **Metacoin**: The default truffle project
+ [HTML reports](https://sc-forks.github.io/metacoin/)
+ [Metacoin with Solcover installed](https://github.com/sc-forks/metacoin) (simple, without configuration)
+ **zeppelin-solidity** at commit 453a19825013a586751b87c67bebd551a252fb50
+ [HTML reports]( https://sc-forks.github.io/zeppelin-solidity/)
+ [Zeppelin with Solcover installed](https://github.com/sc-forks/zeppelin-solidity) (declares own coverage network in truffle.js)
+ **numeraire** at commit 5ac3fa432c6b4192468c95a66e52ca086c804c95
+ [HTML reports](https://sc-forks.github.io/contract/)
+ [Numeraire with Solcover installed](https://github.com/sc-forks/contract) (uses .solcover.js)

###TODO
### TODO

- [ ] **TESTS**
- [ ] Turn into a true command line tool, rather than just a hacked-together script
- [ ] Release on NPM
- [ ] Do not modify the `../contract/` directory at all during operation (might need changes to truffle)
- [ ] Support for arbitrary testing commands
- [ ] [You tell me](http://github.com/JoinColony/solcover/issues)
31 changes: 22 additions & 9 deletions coverageMap.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,7 @@
*/
const SolidityCoder = require('web3/lib/solidity/coder.js');
const path = require('path');

const lineTopic = 'b8995a65f405d9756b41a334f38d8ff0c93c4934e170d3c1429c3e7ca101014d';
const functionTopic = 'd4ce765fd23c5cc3660249353d61ecd18ca60549dd62cb9ca350a4244de7b87f';
const branchTopic = 'd4cf56ed5ba572684f02f889f12ac42d9583c8e3097802060e949bfbb3c1bff5';
const statementTopic = 'b51abbff580b3a34bbc725f2dc6f736e9d4b45a41293fd0084ad865a31fde0c8';
const keccak = require('keccakjs');

/**
* Converts solcover event data into an object that can be
Expand All @@ -20,6 +16,10 @@ module.exports = class CoverageMap {

constructor() {
this.coverage = {};
this.lineTopics = [];
this.functionTopics = [];
this.branchTopics = [];
this.statementTopics = [];
}

/**
Expand All @@ -29,6 +29,7 @@ module.exports = class CoverageMap {
* @param {String} canonicalContractPath target file location
* @return {Object} coverage map with all values set to zero
*/

addContract(info, canonicalContractPath) {
this.coverage[canonicalContractPath] = {
l: {},
Expand Down Expand Up @@ -56,6 +57,17 @@ module.exports = class CoverageMap {
for (let x = 1; x <= Object.keys(info.statementMap).length; x++) {
this.coverage[canonicalContractPath].s[x] = 0;
}

const keccakhex = (x => {
const hash = new keccak(256); // eslint-disable-line new-cap
hash.update(x);
return hash.digest('hex');
});

this.lineTopics.push(keccakhex('__Coverage' + info.contractName + '(string,uint256)'));
this.functionTopics.push(keccakhex('__FunctionCoverage' + info.contractName + '(string,uint256)'));
this.branchTopics.push(keccakhex('__BranchCoverage' + info.contractName + '(string,uint256,uint256)'));
this.statementTopics.push(keccakhex('__StatementCoverage' + info.contractName + '(string,uint256)'));
}

/**
Expand All @@ -68,19 +80,20 @@ module.exports = class CoverageMap {
generate(events, pathPrefix) {
for (let idx = 0; idx < events.length; idx++) {
const event = JSON.parse(events[idx]);
if (event.topics.indexOf(lineTopic) >= 0) {

if (event.topics.filter(t => this.lineTopics.indexOf(t) >= 0).length > 0) {
const data = SolidityCoder.decodeParams(['string', 'uint256'], event.data.replace('0x', ''));
const canonicalContractPath = data[0];
this.coverage[canonicalContractPath].l[data[1].toNumber()] += 1;
} else if (event.topics.indexOf(functionTopic) >= 0) {
} else if (event.topics.filter(t => this.functionTopics.indexOf(t) >= 0).length > 0) {
const data = SolidityCoder.decodeParams(['string', 'uint256'], event.data.replace('0x', ''));
const canonicalContractPath = data[0];
this.coverage[canonicalContractPath].f[data[1].toNumber()] += 1;
} else if (event.topics.indexOf(branchTopic) >= 0) {
} else if (event.topics.filter(t => this.branchTopics.indexOf(t) >= 0).length > 0) {
const data = SolidityCoder.decodeParams(['string', 'uint256', 'uint256'], event.data.replace('0x', ''));
const canonicalContractPath = data[0];
this.coverage[canonicalContractPath].b[data[1].toNumber()][data[2].toNumber()] += 1;
} else if (event.topics.indexOf(statementTopic) >= 0) {
} else if (event.topics.filter(t => this.statementTopics.indexOf(t) >= 0).length > 0) {
const data = SolidityCoder.decodeParams(['string', 'uint256'], event.data.replace('0x', ''));
const canonicalContractPath = data[0];
this.coverage[canonicalContractPath].s[data[1].toNumber()] += 1;
Expand Down
Loading