Skip to content

Commit

Permalink
ipsl: use blocks interface
Browse files Browse the repository at this point in the history
Will be fine performance wise once #57 is fixed.
  • Loading branch information
Jorropo committed Jan 27, 2023
2 parents 180f05a + 4185708 commit 65efc10
Show file tree
Hide file tree
Showing 17 changed files with 297 additions and 122 deletions.
65 changes: 57 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,66 @@
go-libipfs
=======================
<h1 align="center">
go-libipfs 🍌
<br>
<img src="https://raw.githubusercontent.com/ipfs/go-libipfs/main/logo.svg" alt="go-libipfs logo" title="go-libipfs logo" width="200">
<br>
</h1>
<p align="center" style="font-size: 1.2rem;">A library for building IPFS applications and implementations.</p>

> A library for building IPFS implementations
<hr />

Go-libips is a library for building IPFS implementations and tools in Go. It contains reusable functionality useful for interacting and experimenting with IPFS.
[![Go Test](https://github.com/ipfs/go-libipfs/actions/workflows/go-test.yml/badge.svg)](https://github.com/ipfs/go-libipfs/actions/workflows/go-test.yml)
[![Go Docs](https://img.shields.io/badge/godoc-reference-blue.svg)](https://pkg.go.dev/github.com/ipfs/go-libipfs)
[![codecov](https://codecov.io/gh/ipfs/go-libipfs/branch/main/graph/badge.svg?token=9eG7d8fbCB)](https://codecov.io/gh/ipfs/go-libipfs)

This is also used by [Kubo](https://github.com/ipfs/kubo) for its core functionality.
##

Currently this library is a target for consolidating Go IPFS repositories, and will receive minor version releases as repositories are consolidated into it. We are initially focused on merely consolidating repositories, *not* refactoring across packages. Once repositories are mostly consolidated, *then* we will begin refactoring this library holistically. Individual components can still be worked on and refactored individually, but please refrain from trying to refactor across components.
Go-libips is a component library for building IPFS applications and implementations in Go.

## Contributing
Some scenarios in which you may find go-libipfs helpful:

* You are building an application that interacts with the IPFS network
* You are building an IPFS implementation
* You want to reuse some components of IPFS such as its Kademlia DHT, Bitswap, data encoding, etc.
* You want to experiment with IPFS

Go-libipfs powers [Kubo](https://github.com/ipfs/kubo), which is the most popular IPFS implementation, so its code has been battle-tested on the IPFS network for years, and is well-understood by the community.

## What kind of components does go-libipfs have?

Go-libipfs includes high-quality components useful for interacting with IPFS protocols, public and private IPFS networks, and content-addressed data, such as:

- Content routing (DHT, delegated content routing, providing)
- Data transfer (gateways, Bitswap, incremental verification)
- Naming and mutability (name resolution, IPNS)
- Interacting with public and private IPFS networks
- Working with content-addressed data

Go-libipfs aims to provide a cohesive interface into these components. Note that not all of the underlying components necessarily reside in this respository.

## Getting started
TODO

## Should I add my IPFS component to go-libipfs?
We happily accept external contributions! However, go-libipfs maintains a high quality bar, so code accepted into go-libipfs must meet some minimum maintenance criteria:

* Actively maintained
* Must be actively used by, or will be included in software that is actively used by, a significant number of users or production systems. Code that is not actively used cannot be properly maintained.
* Must have multiple engineers who are willing and able to maintain the relevant code in go-libipfs for a long period of time.
* If either of these changes, go-libipfs maintainers will consider removing the component from go-libipfs.
* Adequately tested
* At least with unit tests
* Ideally also including integration tests with other components
* Adequately documented
* Godocs at minimum
* Complex components should have their own doc.go or README.md describing the component, its use cases, tradeoffs, design rationale, etc.
* If the maintainers are not go-libipfs maintainers, then the component must include a CODEOWNERS file with at least two code owners who can commit to reviewing PRs

If you have some experimental component that you think would benefit the IPFS community, we suggest you build the component in your own repository until it's clear that there's community demand for it, and then open an issue in this repository to discuss including it in go-libipfs.

## Help

If you have questions, feel free to open an issue. You can also find the go-libipfs maintainers in [Slack](https://filecoin.io/slack/) at #go-libipfs-maintainers.

Contributions are welcome! This repository is part of the IPFS project and therefore governed by our [contributing guidelines](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md).

## License

Expand Down
82 changes: 82 additions & 0 deletions blocks/blocks.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
// Package blocks contains the lowest level of IPLD data structures.
// A block is raw data accompanied by a CID. The CID contains the multihash
// corresponding to the block.
package blocks

import (
"errors"
"fmt"

cid "github.com/ipfs/go-cid"
u "github.com/ipfs/go-ipfs-util"
mh "github.com/multiformats/go-multihash"
)

// ErrWrongHash is returned when the Cid of a block is not the expected
// according to the contents. It is currently used only when debugging.
var ErrWrongHash = errors.New("data did not match given hash")

// Block provides abstraction for blocks implementations.
type Block interface {
RawData() []byte
Cid() cid.Cid
String() string
Loggable() map[string]interface{}
}

// A BasicBlock is a singular block of data in ipfs. It implements the Block
// interface.
type BasicBlock struct {
cid cid.Cid
data []byte
}

// NewBlock creates a Block object from opaque data. It will hash the data.
func NewBlock(data []byte) *BasicBlock {
// TODO: fix assumptions
return &BasicBlock{data: data, cid: cid.NewCidV0(u.Hash(data))}
}

// NewBlockWithCid creates a new block when the hash of the data
// is already known, this is used to save time in situations where
// we are able to be confident that the data is correct.
func NewBlockWithCid(data []byte, c cid.Cid) (*BasicBlock, error) {
if u.Debug {
chkc, err := c.Prefix().Sum(data)
if err != nil {
return nil, err
}

if !chkc.Equals(c) {
return nil, ErrWrongHash
}
}
return &BasicBlock{data: data, cid: c}, nil
}

// Multihash returns the hash contained in the block CID.
func (b *BasicBlock) Multihash() mh.Multihash {
return b.cid.Hash()
}

// RawData returns the block raw contents as a byte slice.
func (b *BasicBlock) RawData() []byte {
return b.data
}

// Cid returns the content identifier of the block.
func (b *BasicBlock) Cid() cid.Cid {
return b.cid
}

// String provides a human-readable representation of the block CID.
func (b *BasicBlock) String() string {
return fmt.Sprintf("[Block %s]", b.Cid())
}

// Loggable returns a go-log loggable item.
func (b *BasicBlock) Loggable() map[string]interface{} {
return map[string]interface{}{
"block": b.Cid().String(),
}
}
98 changes: 98 additions & 0 deletions blocks/blocks_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
package blocks

import (
"bytes"
"testing"

cid "github.com/ipfs/go-cid"
u "github.com/ipfs/go-ipfs-util"
mh "github.com/multiformats/go-multihash"
)

func TestBlocksBasic(t *testing.T) {

// Test empty data
empty := []byte{}
NewBlock(empty)

// Test nil case
NewBlock(nil)

// Test some data
NewBlock([]byte("Hello world!"))
}

func TestData(t *testing.T) {
data := []byte("some data")
block := NewBlock(data)

if !bytes.Equal(block.RawData(), data) {
t.Error("data is wrong")
}
}

func TestHash(t *testing.T) {
data := []byte("some other data")
block := NewBlock(data)

hash, err := mh.Sum(data, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}

if !bytes.Equal(block.Multihash(), hash) {
t.Error("wrong multihash")
}
}

func TestCid(t *testing.T) {
data := []byte("yet another data")
block := NewBlock(data)
c := block.Cid()

if !bytes.Equal(block.Multihash(), c.Hash()) {
t.Error("key contains wrong data")
}
}

func TestManualHash(t *testing.T) {
oldDebugState := u.Debug
defer (func() {
u.Debug = oldDebugState
})()

data := []byte("I can't figure out more names .. data")
hash, err := mh.Sum(data, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}

c := cid.NewCidV0(hash)

u.Debug = false
block, err := NewBlockWithCid(data, c)
if err != nil {
t.Fatal(err)
}

if !bytes.Equal(block.Multihash(), hash) {
t.Error("wrong multihash")
}

data[5] = byte((uint32(data[5]) + 5) % 256) // Transfrom hash to be different
block, err = NewBlockWithCid(data, c)
if err != nil {
t.Fatal(err)
}

if !bytes.Equal(block.Multihash(), hash) {
t.Error("wrong multihash")
}

u.Debug = true

_, err = NewBlockWithCid(data, c)
if err != ErrWrongHash {
t.Fatal(err)
}
}
2 changes: 2 additions & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
comment:
layout: "reach, diff, files"
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ require (
github.com/gogo/protobuf v1.3.2
github.com/gorilla/mux v1.8.0
github.com/ipfs/go-bitfield v1.0.0
github.com/ipfs/go-block-format v0.0.3
github.com/ipfs/go-blockservice v0.5.0
github.com/ipfs/go-cid v0.3.2
github.com/ipfs/go-datastore v0.6.0
Expand Down Expand Up @@ -46,6 +45,7 @@ require (
github.com/google/uuid v1.3.0 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
github.com/ipfs/go-block-format v0.1.1 // indirect
github.com/ipfs/go-ipfs-ds-help v1.1.0 // indirect
github.com/ipfs/go-ipfs-exchange-interface v0.2.0 // indirect
github.com/ipfs/go-ipld-cbor v0.0.6 // indirect
Expand Down
3 changes: 2 additions & 1 deletion go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,9 @@ github.com/ipfs/go-bitfield v1.0.0 h1:y/XHm2GEmD9wKngheWNNCNL0pzrWXZwCdQGv1ikXkn
github.com/ipfs/go-bitfield v1.0.0/go.mod h1:N/UiujQy+K+ceU1EF5EkVd1TNqevLrCQMIcAEPrdtus=
github.com/ipfs/go-bitswap v0.11.0 h1:j1WVvhDX1yhG32NTC9xfxnqycqYIlhzEzLXG/cU1HyQ=
github.com/ipfs/go-block-format v0.0.2/go.mod h1:AWR46JfpcObNfg3ok2JHDUfdiHRgWhJgCQF+KIgOPJY=
github.com/ipfs/go-block-format v0.0.3 h1:r8t66QstRp/pd/or4dpnbVfXT5Gt7lOqRvC+/dDTpMc=
github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WWgQ8H5l6P8MVk=
github.com/ipfs/go-block-format v0.1.1 h1:129vSO3zwbsYADcyQWcOYiuCpAqt462SFfqFHdFJhhI=
github.com/ipfs/go-block-format v0.1.1/go.mod h1:+McEIT+g52p+zz5xGAABGSOKrzmrdX97bc0USBdWPUs=
github.com/ipfs/go-blockservice v0.5.0 h1:B2mwhhhVQl2ntW2EIpaWPwSCxSuqr5fFA93Ms4bYLEY=
github.com/ipfs/go-blockservice v0.5.0/go.mod h1:W6brZ5k20AehbmERplmERn8o2Ni3ZZubvAxaIUeaT6w=
github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
Expand Down
5 changes: 3 additions & 2 deletions ipsl/compile.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"sync"

"github.com/ipfs/go-cid"
"github.com/ipfs/go-libipfs/blocks"
)

type UnreadableRuneReader interface {
Expand Down Expand Up @@ -338,8 +339,8 @@ type traversalScopeNode struct {
scopeNode
}

func (n traversalScopeNode) Traverse(c cid.Cid, b []byte) ([]CidTraversalPair, error) {
r, err := n.scopeNode.result.(Traversal).Traverse(c, b)
func (n traversalScopeNode) Traverse(b blocks.Block) ([]CidTraversalPair, error) {
r, err := n.scopeNode.result.(Traversal).Traverse(b)
if err != nil {
return nil, err
}
Expand Down
35 changes: 5 additions & 30 deletions ipsl/helpers/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,43 +8,18 @@ import (
"errors"
"fmt"

"github.com/ipfs/go-libipfs/ipsl"

"github.com/ipfs/go-blockservice"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-libipfs/blocks"
"github.com/ipfs/go-libipfs/ipsl"
)

var _ ByteBlockGetter = BlockGetterToByteBlockGetter{}

// BlockGetterToByteBlockGetter implements [ByteBlockGetter] given an [blockservice.BlockGetter].
// This prevent the optimizations [ByteBlockGetter] is trying to do, so you should ideally not use this.
type BlockGetterToByteBlockGetter struct {
BlockGetter blockservice.BlockGetter
}

func (bgtbbg BlockGetterToByteBlockGetter) GetBlock(ctx context.Context, c cid.Cid) ([]byte, error) {
b, err := bgtbbg.BlockGetter.GetBlock(ctx, c)
if err != nil {
return nil, err
}

return b.RawData(), nil
}

// ByteBlockGetter is like [blockservice.BlockGetter] but it does not wrap in an extra block object.
// This is to avoid making too much garbage, passing blocks interface objects arround is moves stuff on the heap.
type ByteBlockGetter interface {
GetBlock(ctx context.Context, c cid.Cid) ([]byte, error)

// TODO: Add GetBlocks when that is needed.
}

var ErrDepthLimitReached = errors.New("safety depth limit reached")

// SyncDFS perform a synchronous recursive depth-first-search.
// It will return [ErrDepthLimitReached] when the safetyDepthLimit is reached.
// It will wrap errors returned by the call back, so use [errors.Is] to test them.
func SyncDFS(ctx context.Context, c cid.Cid, t ipsl.Traversal, getter ByteBlockGetter, safetyDepthLimit uint, callBack func(cid.Cid, []byte) error) error {
func SyncDFS(ctx context.Context, c cid.Cid, t ipsl.Traversal, getter blockservice.BlockGetter, safetyDepthLimit uint, callBack func(blocks.Block) error) error {
if safetyDepthLimit == 0 {
return ErrDepthLimitReached
}
Expand All @@ -55,12 +30,12 @@ func SyncDFS(ctx context.Context, c cid.Cid, t ipsl.Traversal, getter ByteBlockG
return fmt.Errorf("GetBlock: %w", err)
}

err = callBack(c, block)
err = callBack(block)
if err != nil {
return fmt.Errorf("callBack: %w", err)
}

pairs, err := t.Traverse(c, block)
pairs, err := t.Traverse(block)
if err != nil {
return fmt.Errorf("Traversal.Traverse: %w", err)
}
Expand Down
Loading

0 comments on commit 65efc10

Please sign in to comment.