Add ipfs view & ipfs propagate #406
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,17 +1,145 @@ | ||
| # Specification | ||
|
|
||
| The CLI should package several "core" extensions. | ||
| The API these extensions expose is aimed at `node` environments. | ||
| The aim is to provide convenience to power-users & devs. | ||
|
|
||
| ## IPFS `@aragon/ipfs-utils` | ||
|
|
||
| Goals: | ||
|
|
||
| - installing & starting a daemon with the right configuration | ||
| - interacting with local/remote nodes: authentication, configuration, pinning, etc. | ||
| - help setup a "production" node (?) | ||
| - automatically pin anything published to an APM repository (?) | ||
|
There was a problem hiding this comment. I wouldn't really consider these last two part of the flow; for production you're likely to set up your own infra and the last item is very abusable (we actually used to do this but have since moved away to curating the pinning set). |
||
|
|
||
| ### IPFS Binaries | ||
|
|
||
| Local node: | ||
|
|
||
| - ✔️ `aragon ipfs` - Alias for `aragon ipfs start` | ||
|
There was a problem hiding this comment. I've added a bunch of potential changes here after brainstorming a bit about the I think people usually "optimize" their process and run Example
And that There was a problem hiding this comment. I agree, having management commands for the daemon would be great! It'd also solve things like the race condition in #383. |
||
| - `aragon ipfs start` - Start the **IPFS Daemon** | ||
| - Should start in the background and then finish | ||
| - Should start with the recommended configuration from `$HOMEDIR/.aragon/ipfsconfig.json`. E.g.: | ||
|
|
||
| ```json | ||
| { | ||
| "daemonArgs": [ | ||
| "--migrate", | ||
| "--enable-namesys-pubsub" | ||
| ], | ||
| "logsLocation": "$HOMEDIR/.aragon/ipfs-logs" | ||
| } | ||
| ``` | ||
|
|
||
| - Should save `stdout` and `stderr` to files, e.g.: `stdout-${number}.log` in the `logsLocation` | ||
| - Should warn if it's already running | ||
| - Should error (exit code 1) if it cannot start (missing libs/ports taken) | ||
| - Should inform about where the logs are saved and how to stop it | ||
| - Should run `aragon ipfs status` and finish afterwards (optional) | ||
| - `aragon ipfs stop` - Stop the **IPFS Daemon** | ||
| - Should warn if the node was already stopped | ||
| - `aragon ipfs enable-startup` - Start the **IPFS Daemon** automatically at start-up | ||
|
There was a problem hiding this comment. May also make sense to add the ability to clear the IPFS db. Is the plan for us to use our own DB (rather than one the user may already have from a previous IPFS installation)? There was a problem hiding this comment.
👍
Hmm, we could totally do this, but also allow it to be the default directory. I wonder what should be the default.. 🤔 |
||
|
|
||
| Local/remote node: | ||
|
|
||
| - `aragon ipfs status` - Check the configuration of the **IPFS Daemon** | ||
| - Should print if CORS is enabled | ||
| - Should print which Aragon artifacts are pinned | ||
| - Should print number of peers, repository size | ||
| - Should print which ports are used | ||
| - Should print local ip and public ip | ||
| - Should print bootstrapped nodes | ||
| - If it's a local node: | ||
| - Should print whether 'Run at startup' is enabled | ||
| - If it's not configured correctly: | ||
| - Should ask whether to run `aragon ipfs configure` | ||
| - or Should error (exit code 1) | ||
| - `aragon ipfs configure` - Configure the **IPFS Daemon** | ||
| - Should configure CORS | ||
| - Should pin Aragon artifacts (from ipfs with a fallback to http) | ||
| - Should inform the user about advanced configurations with `ipfs config` | ||
| - ✔️ `aragon ipfs view <cid>` - Display metadata about the content, such as size, links, etc. | ||
| - ✔️ `aragon ipfs propagate <cid>` - Request the content and its links at several gateways, making the files | ||
| more distributed within the network | ||
|
|
||
| ### IPFS API | ||
|
|
||
| Connection: | ||
|
|
||
| - ✔️ `ensureConnection` | ||
| - throws if it cannot be established | ||
| - 🚧 auth headers support | ||
| - `start` | ||
|
|
||
| Installation: | ||
|
|
||
| - `isInstalled` | ||
| - `install` | ||
|
|
||
| Configuration: | ||
|
|
||
| - `hasCORSEnabled` | ||
| - `enableCORS` | ||
| - `hasAragonArtifacts` | ||
| - `listAragonArtifacts` | ||
| - `pinAragonArtifacts` | ||
| - `hasAuthEnabled` | ||
| - `enableAuth` | ||
|
|
||
| Data: | ||
|
|
||
| - ✔️ `getMerkleDAG` | ||
| - ✔️ `extractCIDsFromMerkleDAG` | ||
| - ✔️ `propagateFiles` | ||
|
|
||
| Data viz: | ||
|
|
||
| - ✔️ `stringifyMerkleDAG` | ||
|
|
||
| ## Devchain `@aragon/devchain-utils` | ||
|
|
||
| - Pre-bundled because (is tiny?) | ||
|
|
||
| ### Devchain Binaries | ||
|
|
||
| - `aragon devchain` - start ganache with our aragen-generated db | ||
|
|
||
| ### Devchain API | ||
|
|
||
| - `ensureConnection` | ||
| - `hasAragonDeployements` | ||
| - `deployAragon` | ||
| - `start` -- should save stdout and stderr in some files to be outputed by `aragon devchain output` | ||
|
|
||
| ## Web3 `@aragon/web3-utils` | ||
|
|
||
| Note: perhaps this is better suited for `aragonAPI`. | ||
|
|
||
| ### Web3 API | ||
|
|
||
| - `getRecommendedGasLimit` | ||
|
|
||
| By default `web3.js` uses `web3.eth.Contract.method.myMethod().estimateGas()`, but this value can | ||
| differ from the actual gas that will be used, if the contract you are calling depends on the | ||
| blockhash, blocknumber or any other source of randomness making the gas cost nondeterministic. | ||
|
|
||
| E.g.: the [`MiniMeToken`](https://github.com/aragon/aragon-apps/blob/master/shared/minime/contracts/MiniMeToken.sol) | ||
| contract which snapshots balances at certain block numbers. | ||
|
|
||
| Considering the above-mentioned behavior, the CLI should calculate the recommended gas limit | ||
| as follows: | ||
|
|
||
| - `recommendedGas = estimatedGas` (if `estimatedGas > upperGasLimit`) or | ||
| - `recommendedGas = estimatedGas * DEFAULT_GAS_FUZZ_FACTOR` with a maximum value of `upperGasLimit` | ||
|
|
||
| Where: | ||
|
|
||
| - `upperGasLimit = latestBlock.gasLimit * LAST_BLOCK_GAS_LIMIT_FACTOR` | ||
| - `LAST_BLOCK_GAS_LIMIT_FACTOR = 0.95` | ||
| - `DEFAULT_GAS_FUZZ_FACTOR = 1.5` | ||
|
|
||
| See `src/util.js#getRecommendedGasLimit`. | ||
| (This should probably be its own library since `aragon.js` uses it as well) | ||
|
|
||
| Note: The CLI should ignore the `gas` property of the network from truffle config. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,89 +1,9 @@ | ||
| const startCommand = require('./ipfs_cmds/start') | ||
|
|
||
| exports.builder = function(yargs) { | ||
| return yargs.commandDir('ipfs_cmds') | ||
| } | ||
|
|
||
| exports.command = 'ipfs' | ||
| exports.describe = 'Shortcut for aragon ipfs start' | ||
| exports.handler = startCommand.handler |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import TaskList from 'listr' | ||
| // | ||
| import { | ||
| ensureConnection, | ||
| getMerkleDAG, | ||
| extractCIDsFromMerkleDAG, | ||
| propagateFiles, | ||
| } from '../../lib/ipfs' | ||
| import listrOpts from '../../helpers/listr-options' | ||
|
|
||
| exports.command = 'propagate <cid>' | ||
| exports.describe = | ||
| 'Request the content and its links at several gateways, making the files more distributed within the network.' | ||
|
|
||
| exports.builder = yargs => { | ||
| return yargs.positional('cid', { | ||
| description: 'A self-describing content-addressed identifier', | ||
| }) | ||
| } | ||
|
|
||
| exports.task = ({ apmOptions, silent, debug, cid }) => { | ||
| return new TaskList( | ||
| [ | ||
| { | ||
| title: 'Connect to IPFS', | ||
| task: async ctx => { | ||
| ctx.ipfs = await ensureConnection(apmOptions.ipfs.rpc) | ||
| }, | ||
| }, | ||
| { | ||
| title: 'Fetch the links', | ||
| task: async ctx => { | ||
| ctx.data = await getMerkleDAG(ctx.ipfs.client, cid, { | ||
| recursive: true, | ||
| }) | ||
| }, | ||
| }, | ||
| { | ||
| title: 'Query gateways', | ||
| task: async (ctx, task) => { | ||
| ctx.CIDs = extractCIDsFromMerkleDAG(ctx.data, { | ||
| recursive: true, | ||
| }) | ||
|
|
||
| const logger = text => (task.output = text) | ||
| ctx.result = await propagateFiles(ctx.CIDs, logger) | ||
| }, | ||
| }, | ||
| ], | ||
| listrOpts(silent, debug) | ||
| ) | ||
| } | ||
|
|
||
| exports.handler = async function({ | ||
| reporter, | ||
| apm: apmOptions, | ||
| cid, | ||
| debug, | ||
| silent, | ||
| }) { | ||
| const task = await exports.task({ | ||
| reporter, | ||
| apmOptions, | ||
| cid, | ||
| debug, | ||
| silent, | ||
| }) | ||
|
|
||
| const ctx = await task.run() | ||
|
|
||
| reporter.info( | ||
| `Queried ${ctx.CIDs.length} CIDs at ${ctx.result.gateways.length} gateways` | ||
| ) | ||
| reporter.info(`Requests succeeded: ${ctx.result.succeeded}`) | ||
| reporter.info(`Requests failed: ${ctx.result.failed}`) | ||
| reporter.debug(`Gateways: ${ctx.result.gateways.join(', ')}`) | ||
| reporter.debug( | ||
| `Errors: \n${ctx.result.errors.map(JSON.stringify).join('\n')}` | ||
| ) | ||
| // TODO add your own gateways | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For extensions related to the cli, perhaps it would make sense to prefix them with
cli(so this would become@aragon/cli-ipfs-utils)?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, makes sense.
What about
dao,apm, etc? I think we can publish those without theutilssuffix, e.g.:@aragon/cli-dao.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like
@aragon/cli-<package>format