-
Notifications
You must be signed in to change notification settings - Fork 79
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add ipfs view & ipfs propagate (#406)
- Loading branch information
1 parent
8690a49
commit d4acbc6
Showing
12 changed files
with
552 additions
and
101 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,17 +1,145 @@ | ||
| ## Gas limit | ||
| # Specification | ||
|
|
||
| By default `web3.js` uses `web3.eth.Contract.method.myMethod().estimateGas()`, but this value can differ from the | ||
| actual gas that will be used, if the contract you are calling depends on the blockhash, blocknumber or any other source of | ||
| randomness making the gas cost nondeterministic. | ||
| The CLI should package several "core" extensions. | ||
| The API these extensions expose is aimed at `node` environments. | ||
| The aim is to provide convenience to power-users & devs. | ||
|
|
||
| E.g.: the [`MiniMeToken`](https://github.com/aragon/aragon-apps/blob/master/shared/minime/contracts/MiniMeToken.sol) contract which snapshots balances at certain block numbers. | ||
| ## IPFS `@aragon/ipfs-utils` | ||
|
|
||
| Considering the above-mentioned behavior, the CLI should calculate the recommended gas limit as follows: | ||
| - `recommendedGas = estimatedGas`, if `estimatedGas > upperGasLimit` or | ||
| Goals: | ||
|
|
||
| - installing & starting a daemon with the right configuration | ||
| - interacting with local/remote nodes: authentication, configuration, pinning, etc. | ||
| - help setup a "production" node (?) | ||
| - automatically pin anything published to an APM repository (?) | ||
|
|
||
| ### IPFS Binaries | ||
|
|
||
| Local node: | ||
|
|
||
| - ✔️ `aragon ipfs` - Alias for `aragon ipfs start` | ||
| - `aragon ipfs start` - Start the **IPFS Daemon** | ||
| - Should start in the background and then finish | ||
| - Should start with the recommended configuration from `$HOMEDIR/.aragon/ipfsconfig.json`. E.g.: | ||
|
|
||
| ```json | ||
| { | ||
| "daemonArgs": [ | ||
| "--migrate", | ||
| "--enable-namesys-pubsub" | ||
| ], | ||
| "logsLocation": "$HOMEDIR/.aragon/ipfs-logs" | ||
| } | ||
| ``` | ||
|
|
||
| - Should save `stdout` and `stderr` to files, e.g.: `stdout-${number}.log` in the `logsLocation` | ||
| - Should warn if it's already running | ||
| - Should error (exit code 1) if it cannot start (missing libs/ports taken) | ||
| - Should inform about where the logs are saved and how to stop it | ||
| - Should run `aragon ipfs status` and finish afterwards (optional) | ||
| - `aragon ipfs stop` - Stop the **IPFS Daemon** | ||
| - Should warn if the node was already stopped | ||
| - `aragon ipfs enable-startup` - Start the **IPFS Daemon** automatically at start-up | ||
|
|
||
| Local/remote node: | ||
|
|
||
| - `aragon ipfs status` - Check the configuration of the **IPFS Daemon** | ||
| - Should print if CORS is enabled | ||
| - Should print which Aragon artifacts are pinned | ||
| - Should print number of peers, repository size | ||
| - Should print which ports are used | ||
| - Should print local ip and public ip | ||
| - Should print bootstrapped nodes | ||
| - If it's a local node: | ||
| - Should print whether 'Run at startup' is enabled | ||
| - If it's not configured correctly: | ||
| - Should ask whether to run `aragon ipfs configure` | ||
| - or Should error (exit code 1) | ||
| - `aragon ipfs configure` - Configure the **IPFS Daemon** | ||
| - Should configure CORS | ||
| - Should pin Aragon artifacts (from ipfs with a fallback to http) | ||
| - Should inform the user about advanced configurations with `ipfs config` | ||
| - ✔️ `aragon ipfs view <cid>` - Display metadata about the content, such as size, links, etc. | ||
| - ✔️ `aragon ipfs propagate <cid>` - Request the content and its links at several gateways, making the files | ||
| more distributed within the network | ||
|
|
||
| ### IPFS API | ||
|
|
||
| Connection: | ||
|
|
||
| - ✔️ `ensureConnection` | ||
| - throws if it cannot be established | ||
| - 🚧 auth headers support | ||
| - `start` | ||
|
|
||
| Installation: | ||
|
|
||
| - `isInstalled` | ||
| - `install` | ||
|
|
||
| Configuration: | ||
|
|
||
| - `hasCORSEnabled` | ||
| - `enableCORS` | ||
| - `hasAragonArtifacts` | ||
| - `listAragonArtifacts` | ||
| - `pinAragonArtifacts` | ||
| - `hasAuthEnabled` | ||
| - `enableAuth` | ||
|
|
||
| Data: | ||
|
|
||
| - ✔️ `getMerkleDAG` | ||
| - ✔️ `extractCIDsFromMerkleDAG` | ||
| - ✔️ `propagateFiles` | ||
|
|
||
| Data viz: | ||
|
|
||
| - ✔️ `stringifyMerkleDAG` | ||
|
|
||
| ## Devchain `@aragon/devchain-utils` | ||
|
|
||
| - Pre-bundled because (is tiny?) | ||
|
|
||
| ### Devchain Binaries | ||
|
|
||
| - `aragon devchain` - start ganache with our aragen-generated db | ||
|
|
||
| ### Devchain API | ||
|
|
||
| - `ensureConnection` | ||
| - `hasAragonDeployements` | ||
| - `deployAragon` | ||
| - `start` -- should save stdout and stderr in some files to be outputed by `aragon devchain output` | ||
|
|
||
| ## Web3 `@aragon/web3-utils` | ||
|
|
||
| Note: perhaps this is better suited for `aragonAPI`. | ||
|
|
||
| ### Web3 API | ||
|
|
||
| - `getRecommendedGasLimit` | ||
|
|
||
| By default `web3.js` uses `web3.eth.Contract.method.myMethod().estimateGas()`, but this value can | ||
| differ from the actual gas that will be used, if the contract you are calling depends on the | ||
| blockhash, blocknumber or any other source of randomness making the gas cost nondeterministic. | ||
|
|
||
| E.g.: the [`MiniMeToken`](https://github.com/aragon/aragon-apps/blob/master/shared/minime/contracts/MiniMeToken.sol) | ||
| contract which snapshots balances at certain block numbers. | ||
|
|
||
| Considering the above-mentioned behavior, the CLI should calculate the recommended gas limit | ||
| as follows: | ||
|
|
||
| - `recommendedGas = estimatedGas` (if `estimatedGas > upperGasLimit`) or | ||
| - `recommendedGas = estimatedGas * DEFAULT_GAS_FUZZ_FACTOR` with a maximum value of `upperGasLimit` | ||
|
|
||
| Where `upperGasLimit = latestBlock.gasLimit * LAST_BLOCK_GAS_LIMIT_FACTOR`, `DEFAULT_GAS_FUZZ_FACTOR = 1.5` and `LAST_BLOCK_GAS_LIMIT_FACTOR = 0.95`. | ||
| Where: | ||
|
|
||
| - `upperGasLimit = latestBlock.gasLimit * LAST_BLOCK_GAS_LIMIT_FACTOR` | ||
| - `LAST_BLOCK_GAS_LIMIT_FACTOR = 0.95` | ||
| - `DEFAULT_GAS_FUZZ_FACTOR = 1.5` | ||
|
|
||
| See `src/util.js#getRecommendedGasLimit`. (This should probably be its own library) | ||
| See `src/util.js#getRecommendedGasLimit`. | ||
| (This should probably be its own library since `aragon.js` uses it as well) | ||
|
|
||
| The CLI should ignore the `gas` property of the network from truffle config. | ||
| Note: The CLI should ignore the `gas` property of the network from truffle config. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,89 +1,9 @@ | ||
| const path = require('path') | ||
| const TaskList = require('listr') | ||
| const { | ||
| startIPFSDaemon, | ||
| isIPFSCORS, | ||
| setIPFSCORS, | ||
| isIPFSRunning, | ||
| } = require('../helpers/ipfs-daemon') | ||
| const startCommand = require('./ipfs_cmds/start') | ||
|
|
||
| const IPFS = require('ipfs-api') | ||
| const listrOpts = require('../helpers/listr-options') | ||
|
|
||
| exports.command = 'ipfs' | ||
|
|
||
| exports.describe = 'Start IPFS daemon configured to work with Aragon' | ||
|
|
||
| exports.task = ({ apmOptions, silent, debug }) => { | ||
| return new TaskList( | ||
| [ | ||
| { | ||
| title: 'Start IPFS', | ||
| task: async (ctx, task) => { | ||
| // If the dev manually set their IPFS node, skip install and running check | ||
| if (apmOptions.ipfs.rpc.default) { | ||
| const running = await isIPFSRunning(apmOptions.ipfs.rpc) | ||
| if (!running) { | ||
| task.output = 'Starting IPFS at port: ' + apmOptions.ipfs.rpc.port | ||
| await startIPFSDaemon() | ||
| ctx.started = true | ||
| await setIPFSCORS(apmOptions.ipfs.rpc) | ||
| } else { | ||
| task.output = 'IPFS is started, checking CORS config' | ||
| await setIPFSCORS(apmOptions.ipfs.rpc) | ||
| return ( | ||
| 'Connected to IPFS daemon at port: ' + apmOptions.ipfs.rpc.port | ||
| ) | ||
| } | ||
| } else { | ||
| await isIPFSCORS(apmOptions.ipfs.rpc) | ||
| return 'Connecting to provided IPFS daemon' | ||
| } | ||
| }, | ||
| }, | ||
| { | ||
| title: 'Add local files', | ||
| task: ctx => { | ||
| const ipfs = IPFS('localhost', '5001', { protocol: 'http' }) | ||
| const files = path.resolve( | ||
| require.resolve('@aragon/aragen'), | ||
| '../ipfs-cache' | ||
| ) | ||
|
|
||
| return new Promise((resolve, reject) => { | ||
| ipfs.util.addFromFs( | ||
| files, | ||
| { recursive: true, ignore: 'node_modules' }, | ||
| (err, files) => { | ||
| if (err) return reject(err) | ||
| resolve(files) | ||
| } | ||
| ) | ||
| }) | ||
| }, | ||
| }, | ||
| ], | ||
| listrOpts(silent, debug) | ||
| ) | ||
| exports.builder = function(yargs) { | ||
| return yargs.commandDir('ipfs_cmds') | ||
| } | ||
|
|
||
| exports.handler = function({ reporter, apm: apmOptions }) { | ||
| const task = exports.task({ apmOptions }) | ||
|
|
||
| task | ||
| .run() | ||
| .then(ctx => { | ||
| if (ctx.started) { | ||
| reporter.info( | ||
| 'IPFS daemon is now running. Stopping this process will stop IPFS' | ||
| ) | ||
| } else { | ||
| reporter.warning('Didnt start IPFS, port busy') | ||
| process.exit() | ||
| } | ||
| }) | ||
| .catch(err => { | ||
| reporter.error(err) | ||
| process.exit(1) | ||
| }) | ||
| } | ||
| exports.command = 'ipfs' | ||
| exports.describe = 'Shortcut for aragon ipfs start' | ||
| exports.handler = startCommand.handler |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import TaskList from 'listr' | ||
| // | ||
| import { | ||
| ensureConnection, | ||
| getMerkleDAG, | ||
| extractCIDsFromMerkleDAG, | ||
| propagateFiles, | ||
| } from '../../lib/ipfs' | ||
| import listrOpts from '../../helpers/listr-options' | ||
|
|
||
| exports.command = 'propagate <cid>' | ||
| exports.describe = | ||
| 'Request the content and its links at several gateways, making the files more distributed within the network.' | ||
|
|
||
| exports.builder = yargs => { | ||
| return yargs.positional('cid', { | ||
| description: 'A self-describing content-addressed identifier', | ||
| }) | ||
| } | ||
|
|
||
| exports.task = ({ apmOptions, silent, debug, cid }) => { | ||
| return new TaskList( | ||
| [ | ||
| { | ||
| title: 'Connect to IPFS', | ||
| task: async ctx => { | ||
| ctx.ipfs = await ensureConnection(apmOptions.ipfs.rpc) | ||
| }, | ||
| }, | ||
| { | ||
| title: 'Fetch the links', | ||
| task: async ctx => { | ||
| ctx.data = await getMerkleDAG(ctx.ipfs.client, cid, { | ||
| recursive: true, | ||
| }) | ||
| }, | ||
| }, | ||
| { | ||
| title: 'Query gateways', | ||
| task: async (ctx, task) => { | ||
| ctx.CIDs = extractCIDsFromMerkleDAG(ctx.data, { | ||
| recursive: true, | ||
| }) | ||
|
|
||
| const logger = text => (task.output = text) | ||
| ctx.result = await propagateFiles(ctx.CIDs, logger) | ||
| }, | ||
| }, | ||
| ], | ||
| listrOpts(silent, debug) | ||
| ) | ||
| } | ||
|
|
||
| exports.handler = async function({ | ||
| reporter, | ||
| apm: apmOptions, | ||
| cid, | ||
| debug, | ||
| silent, | ||
| }) { | ||
| const task = await exports.task({ | ||
| reporter, | ||
| apmOptions, | ||
| cid, | ||
| debug, | ||
| silent, | ||
| }) | ||
|
|
||
| const ctx = await task.run() | ||
|
|
||
| reporter.info( | ||
| `Queried ${ctx.CIDs.length} CIDs at ${ctx.result.gateways.length} gateways` | ||
| ) | ||
| reporter.info(`Requests succeeded: ${ctx.result.succeeded}`) | ||
| reporter.info(`Requests failed: ${ctx.result.failed}`) | ||
| reporter.debug(`Gateways: ${ctx.result.gateways.join(', ')}`) | ||
| reporter.debug( | ||
| `Errors: \n${ctx.result.errors.map(JSON.stringify).join('\n')}` | ||
| ) | ||
| // TODO add your own gateways | ||
| } |
Oops, something went wrong.