Skip to content
This repository has been archived by the owner on Aug 11, 2021. It is now read-only.

refactor: pass in IPLD Formats into the constructor #164

Merged
merged 1 commit into from
Oct 25, 2018
Merged

refactor: pass in IPLD Formats into the constructor #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ Want to get started? Check our examples folder. You can check the development st
- [`.remove(cid, callback)`](#removecid-callback)
- [`.support.add(multicodec, formatResolver, formatUtil)`](#supportaddmulticodec-formatresolver-formatutil)
- [`.support.rm(multicodec)`](#supportrmmulticodec)
- [Properties](#properties)
- [`defaultOptions`](#defaultoptions)
- [Packages](#packages)
- [Contribute](#contribute)
- [License](#license)
Expand Down Expand Up @@ -115,6 +117,23 @@ const blockService = new IpfsBlockService(repo)
const ipld = new Ipld({blockService: blockService})
```

##### `options.formats`

| Type | Default |
|------|---------|
| Array of [IPLD Format](https://github.com/ipld/interface-ipld-format) implementations | `[require('ipld-dag-cbor'), require('ipld-dag-pb'), require('ipld-raw')]` |

By default only the [dag-cbor](https://github.com/ipld/js-ipld-dag-cbor)), [dag-pb](https://github.com/ipld/js-ipld-dag-pb)) and [raw](https://github.com/ipld/js-ipld-raw)) IPLD Formats are supported. Other formats need to be added manually. Here is an example if you want to have support for [ipld-git](https://github.com/ipld/js-ipld-git) only:

```js
const ipldGit = require('ipld-git')

const ipld = new Ipld({
formats: [ipldGit],
…
})
```

### `.put(node, options, callback)`

> Store the given node of a recognized IPLD Format.
Expand Down Expand Up @@ -161,6 +180,12 @@ const ipld = new Ipld({blockService: blockService})

> Removes support of an IPLD Format

### Properties

#### `defaultOptions`

> Default options for IPLD.

## Packages

Listing of dependencies from the IPLD ecosystem.
Expand Down
14 changes: 9 additions & 5 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,21 @@
"license": "MIT",
"devDependencies": {
"aegir": "^15.2.0",
"bitcoinjs-lib": "^4.0.2",
"chai": "^4.1.2",
"dirty-chai": "^2.0.1",
"eth-hash-to-cid": "~0.1.1",
"ethereumjs-block": "^2.0.1",
"ipld-bitcoin": "~0.1.7",
"ipld-ethereum": "^2.0.1",
"ipld-git": "~0.2.1",
"ipld-zcash": "~0.1.6",
"merkle-patricia-tree": "^2.3.2",
"multihashes": "~0.4.14",
"ncp": "^2.0.0",
"rimraf": "^2.6.2",
"rlp": "^2.1.0"
"rlp": "^2.1.0",
"zcash-bitcore-lib": "~0.13.20-rc3"
},
"dependencies": {
"async": "^2.6.1",
Expand All @@ -50,13 +57,10 @@
"ipfs-block": "~0.7.1",
"ipfs-block-service": "~0.14.0",
"ipfs-repo": "~0.24.0",
"ipld-bitcoin": "~0.1.7",
"ipld-dag-cbor": "~0.13.0",
"ipld-dag-pb": "~0.14.10",
"ipld-ethereum": "^2.0.1",
"ipld-git": "~0.2.1",
"ipld-raw": "^2.0.1",
"ipld-zcash": "~0.1.6",
"merge-options": "^1.0.1",
"pull-defer": "~0.2.3",
"pull-stream": "^3.6.9",
"pull-traverse": "^1.0.3"
Expand Down
81 changes: 24 additions & 57 deletions src/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,73 +14,26 @@ const map = require('async/map')
const series = require('async/series')
const waterfall = require('async/waterfall')
const MemoryStore = require('interface-datastore').MemoryDatastore
const mergeOptions = require('merge-options')
const ipldDagCbor = require('ipld-dag-cbor')
const ipldDagPb = require('ipld-dag-pb')
const ipldRaw = require('ipld-raw')

function noop () {}

class IPLDResolver {
constructor (options) {
constructor (userOptions) {
const options = mergeOptions(IPLDResolver.defaultOptions, userOptions)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be enough here right ?

Suggested change
const options = mergeOptions(IPLDResolver.defaultOptions, userOptions)
const options = Object.assign({}, IPLDResolver.defaultOptions, userOptions)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want a to have a deep copy. Example:

'use strict'

const mergeOptions = require('merge-options')

const oa1 = {main: {sub: 1}}
const oa2 = {main: {sub: 2}}
const oa = Object.assign({}, oa1, oa2)
oa2.main.sub = 3
console.log('oa:', oa)

const mo1 = {main: {sub: 1}}
const mo2 = {main: {sub: 2}}
const mo = mergeOptions(mo1, mo2)
mo2.main.sub = 3
console.log('mo:',  mo)

Output is:

oa: { main: { sub: 3 } }
mo: { main: { sub: 2 } }

See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/assign#Deep_Clone for more information about Object.assign().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok just wanted to make sure we really need deep copy here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K, cool. I prefer code which doesn't lead to weird surprises in the long run (+ I've coded functional programming language for a looong tim)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vmx can i suggest https://www.npmjs.com/package/assign-deep instead of merge-options. Check https://codesandbox.io/s/742p0rjvy0, assign-deep is smaller and handles non-plain-objects values better like promises (merge-options gives empty object when you should get a Promise). We should use assign-deep everywhere.

Copy link
Member Author

@vmx vmx Oct 25, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hugomrdias: I want to have a deep copy, that makes things easier to debug/reason about. assign-deep does only a shallow copy.

Though I'd love to see that we agree on something decent across all libs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just read ipfs/js-ipfs#1663 (comment). Perhaps it's not always a good idea to copy things. I think I need to put more thought into this. Though I'll merge this for now to keep things going.


if (!options.blockService) {
throw new Error('Missing blockservice')
}

this.bs = options.blockService

// Support by default dag-pb, dag-cbor, git, and eth-*
this.resolvers = {
get 'dag-pb' () {
const format = require('ipld-dag-pb')
return { resolver: format.resolver, util: format.util }
},
get 'dag-cbor' () {
const format = require('ipld-dag-cbor')
return { resolver: format.resolver, util: format.util }
},
get 'git-raw' () {
const format = require('ipld-git')
return { resolver: format.resolver, util: format.util }
},
get 'bitcoin-block' () {
const format = require('ipld-bitcoin')
return { resolver: format.resolver, util: format.util }
},
get 'eth-account-snapshot' () {
const format = require('ipld-ethereum').ethAccountSnapshot
return { resolver: format.resolver, util: format.util }
},
get 'eth-block' () {
const format = require('ipld-ethereum').ethBlock
return { resolver: format.resolver, util: format.util }
},
get 'eth-block-list' () {
const format = require('ipld-ethereum').ethBlockList
return { resolver: format.resolver, util: format.util }
},
get 'eth-state-trie' () {
const format = require('ipld-ethereum').ethStateTrie
return { resolver: format.resolver, util: format.util }
},
get 'eth-storage-trie' () {
const format = require('ipld-ethereum').ethStorageTrie
return { resolver: format.resolver, util: format.util }
},
get 'eth-tx' () {
const format = require('ipld-ethereum').ethTx
return { resolver: format.resolver, util: format.util }
},
get 'eth-tx-trie' () {
const format = require('ipld-ethereum').ethTxTrie
return { resolver: format.resolver, util: format.util }
},
get raw () {
const format = require('ipld-raw')
return { resolver: format.resolver, util: format.util }
},
get 'zcash-block' () {
const format = require('ipld-zcash')
return { resolver: format.resolver, util: format.util }
}
}
// Object with current list of active resolvers
this.resolvers = {}

// API entry point
this.support = {}

// Adds support for an IPLD format
Expand All @@ -100,6 +53,13 @@ class IPLDResolver {
delete this.resolvers[multicodec]
}
}

// Enable all supplied formats
for (const format of options.formats) {
const {resolver, util} = format
const multicodec = resolver.multicodec
this.support.add(multicodec, resolver, util)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll undo the perf work here #145 as IPFS (and potentially others) will have to require these formats upfront.

Can we load these on demand? This would actually be good for IPFS in browsers because we could exclude the implementations from the bundle but add loaders to load in the code when/if needed.

Something like this?:

// In IPFS:
ipld.support.add('git-raw', () => new Promise((resolve, reject) => {
  // Could replace with a XHR request. In Node.js we just:
  resolve(require('ipld-git'))
}))
// In ipld
let format
if (typeof this.support[codec] === 'function') {
  format = await this.support[codec]()
  this.support[codec] = format
} else if (this.support[codec]) {
  format = this.support[codec]
} else {
  throw new Error('missing format')
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea was that you'd normally pass in only the things you need anyway. But then I found out that this is certainly not the case for the js-ifps CLI, where you want support for all formats.

That's a good idea. Thanks for also flushing out the code.

}

get (cid, path, options, callback) {
Expand Down Expand Up @@ -414,6 +374,13 @@ class IPLDResolver {
}
}

/**
* Default options for IPLD.
*/
IPLDResolver.defaultOptions = {
formats: [ipldDagCbor, ipldDagPb, ipldRaw]
}

/**
* Create an IPLD resolver with an inmemory blockservice and
* repo.
Expand Down
5 changes: 4 additions & 1 deletion test/ipld-bitcoin.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,10 @@ module.exports = (repo) => {

before((done) => {
const bs = new BlockService(repo)
resolver = new IPLDResolver({blockService: bs})
resolver = new IPLDResolver({
blockService: bs,
formats: [ipldBitcoin]
})

series([
(cb) => {
Expand Down
5 changes: 4 additions & 1 deletion test/ipld-eth-block.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@ module.exports = (repo) => {

before((done) => {
const bs = new BlockService(repo)
resolver = new IPLDResolver({blockService: bs})
resolver = new IPLDResolver({
blockService: bs,
formats: [ipldEthBlock]
})

series([
(cb) => {
Expand Down
7 changes: 6 additions & 1 deletion test/ipld-eth.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ const expect = chai.expect
chai.use(dirtyChai)
const rlp = require('rlp')
const BlockService = require('ipfs-block-service')
const ipldEthBlock = require('ipld-ethereum').ethBlock
const ipldEthStateTrie = require('ipld-ethereum').ethStateTrie
const loadFixture = require('aegir/fixtures')
const async = require('async')
const cidForHash = require('eth-hash-to-cid')
Expand All @@ -24,7 +26,10 @@ module.exports = (repo) => {
before(function (done) {
this.timeout(10 * 1000)
const bs = new BlockService(repo)
resolver = new IPLDResolver({blockService: bs})
resolver = new IPLDResolver({
blockService: bs,
formats: [ipldEthBlock, ipldEthStateTrie]
})

async.waterfall([
readFilesFixture,
Expand Down
5 changes: 4 additions & 1 deletion test/ipld-git.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,10 @@ module.exports = (repo) => {
before((done) => {
const bs = new BlockService(repo)

resolver = new IPLDResolver({blockService: bs})
resolver = new IPLDResolver({
blockService: bs,
formats: [ipldGit]
})

series([
(cb) => {
Expand Down
5 changes: 4 additions & 1 deletion test/ipld-zcash.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,10 @@ module.exports = (repo) => {

before((done) => {
const bs = new BlockService(repo)
resolver = new IPLDResolver({blockService: bs})
resolver = new IPLDResolver({
blockService: bs,
formats: [ipldZcash]
})

series([
(cb) => {
Expand Down