OKRs - 2019 Q2 Go Core Dev

[momack2]

Go IPFS Core Dev OKRs

It's time for us to work on our OKRs for Q2 2019. We should keep in mind our WG goals as expressed in our roadmap. We'll use this PR for proposals and discussions of OKRs the next quarter.

ref #902

• Score 2019 Q1 OKRs
• Async Retrospective
• 2019 Q2 OKRs Open Planning (this thread)
• Move 2019 Q2 OKRs to 2019 Q2 OKRs Spreadsheet

[scout]

asks from the Infra team:

• for the benefit of the gateway, Towards better tracking of gateway performance kubo#5783 needs to be addressed. It would be great to have a member of the go-ipfs team be the DRI for instrumenting the code base, specifically for calls to the http handler
• for the benefit of container deployments, go-ipfs needs to be configurable via environment variables

[momack2]

From @Stebalien in OKR sheet:

go-ipfs meets performance and scalability requirements for Package Managers

• It's possible to "rsync" to go-ipfs (initiative: Writable FUSE)
• Implement provider strategy such that a user can add (and provide) npm or tr-wikipedia without turning off providing and without significantly impacting finding content (initiative: provider subsystem)
• When connected to the global internet, IPNS record resolution takes less than 1s (initiative: IPNS over DNS, IPNS over pubsub, etc.)
• go-ipfs saturates a 100MiB link when downloading a sufficiently large dataset.
• go-ipfs has at most 10% overhead when downloading a large dataset (not including providing).
• go-ipfs has experimental support for modification times and other metadata (initiative: UnixFSv2)

Support IPFS Camp

• Users can reliably find content via the gateway (initiatives: provider subsystem, libp2p stuff)
• Create 2 workshops.

Improve Contributor Experience

• GitCop is removed from go-ipfs (initiative: migrate to MIT + Apache 2.0)
• go-ipfs uses go-ipld-prime
• Bug tracking? Triaging? Responses? Documentation? CoreAPI? Plugins?

Support IPFS Users

• Filecoin has a long-term solution for syncing deep graphs

@magik6k @djdv @Kubuxu @eingenito @hannahhoward @michaelavila @aschmahmann anything to add?

@hsanjuan @lanzafame - I think there's also room for us to help advise you on a better design for the pinning strategy you need for cluster sharding, but we won't have time to implement it ourselves this quarter.

[aschmahmann]

As I mentioned, I'd like more clarification on the performance requirements for IPNS. I've run into scenarios where even running a DHT findpeers operation takes more than 1 second, which may preclude any libp2p-based truly decentralized solution (i.e. requires something like DNS or libp2p super-peers). Adding more specific network requirements that could be emulated by the testlab and benchmarked against would likely be much more helpful to us.

@Stebalien any thoughts?

[scout]

Users can reliably find content via the gateway (initiatives: provider subsystem, libp2p stuff)

What does reliably mean here? Can we define specific performance metrics to meet?

[olizilla]

It'd be rad to get per CID bandwidth stats as per: ipfs/kubo#4740 so we could show things like share-ratios in npm-on-ipfs for modules you are co-hosting.

[Stebalien]

As I mentioned, I'd like more clarification on the performance requirements for IPNS. I've run into scenarios where even running a DHT findpeers operation takes more than 1 second, which may preclude any libp2p-based truly decentralized solution (i.e. requires something like DNS or libp2p super-peers). Adding more specific network requirements that could be emulated by the testlab and benchmarked against would likely be much more helpful to us.

I agree. For the moment at least, we're not going to get fast IPNS if we depend on the DHT.

It'd be rad to get per CID bandwidth stats as per: ipfs/kubo#4740 so we could show things like share-ratios in npm-on-ipfs for modules you are co-hosting.

While I agree that would be cool, I feel that motivating this with package managers is a bit of a stretch.

[olizilla]

I politely disagree. If we don't have capacity to tackle the bw per cid stats, that's fine, but my motives are genuine.

[andrew]

Collecting stats for package managers is definitely going to be important given that using IPFS will remove the traditional download stats that many pms use for decision making, promotion and discovery, more notes here: ipfs-inactive/package-managers#6

[Stebalien]

@andrew unfortunately, I'm not sure if we can collect accurate metrics there. However, I we could get decent relative metrics (probabilistically).

[momack2]

More from our meeting:

• There is a way to ingest files into IPFS quickly for very large files and many many small files (less than 2x over cp)
• Goal: down to 500 open issues on go-ipfs by end of quarter
• document what types of issues should live in go-ipfs and migrate open issues that don't fit those guidelines to the appropriate place (design issues to ipfs/ipfs or ipfs/notes, questions to forum, close issues more (old bugs)! etc)
• Triage tuesdays to chip away at this

@scout - can you describe the problems behind the environment variables and instrumentation ask? We think there are some other things to explore around docker image configs that would be better than adding more environment variables

[Stebalien]

What if cluster, pinning services, the gateways, etc. used go-ipfs as a library? We're getting pretty close to this being a reality and it would allow all these services to hook in and replace portions.

For example, this would make it much easier to try out a new pinning/gc system and/or create custom gateways. Unfortunately, I'm not sure if this is viable for cluster as, at the moment at least, one can incorporate normal go-ipfs nodes into a cluster.

My concern is that there's a pretty high bar for modifying the internals of go-ipfs. We can throw experiments on-top but deeper modifications need to be weighed against the maintenance burden and the potential effects on future changes. However, extension points, events, etc. (things we could add to a go-ipfs as a library) are a lot less risky.

@hsanjuan? @scout?

[scout]

describe the problems behind the environment variables and instrumentation ask? We think there are some other things to explore around docker image configs that would be better than adding more environment variables

We've (infra) kicked around that idea as well. If the go-ipfs team is against more environment variables then we will go the entrypoint scripty route.

Re: instrumentation - Infra is planning to reorganize to take on some of the initial metrics work (using opencensus to be inline with cluster and libp2p) specifically for gateway calls. Is this agreeable to the go-ipfs team?

What if cluster, pinning services, the gateways, etc. used go-ipfs as a library?

I think I'm for it, but would like to talk more about implications

[hannahhoward]

Relevant to this discussion: ipfs/kubo#6208

This is about GraphSync integration in IPFS itself, rather than Filecoin. It's arguably a lower level priority. However, two things:

1. GraphSync could unlock a number of capabilities for the package manager use case (most specifically, fast deep path queeries)
2. GraphSync integration with UnixFS would probably require writing UnixFS V2, which might raise it up the priority list since it becomes a blocker to GraphSync integration, as well as something needed for package managers

[hannahhoward]

Also, I volunteered to DRI the bitswap related OKRs, graphsync OKR , go-ipld-prime OKR, and also potentially the UnixFS V2 related OKR (since I know go-ipld-prime well). However, I think this might be a lot :P Open to suggestions.

[djdv]

@hannahhoward
I'm interested in working on UnixFS V2 if you want to drop that one. We can collaborate around ufs v2 and go-ipld-prime at the same time.

[momack2]

Added more items to the sheet and started assigning to people! Folks - take a look and see if this documents the work correctly?

For this quarter - @Stebalien and I propose we turn all KRs into tracking Github issues and "assign" to appropriate owners (maybe with a special "okrs" label so they're easy to find). That way, if you find you don't have time for something mid-quarter you can unassign and solicit a new owner to grab the issue from you. For any that already have tracking issues, add them to the notes column!