Support private IPFS networks (via swarm.key)

[ianconsolata]

Is your feature request related to a problem? Please describe.
Yes! Private IPFS networks, which are supported in kubo, are not currently enabled in IPFS Desktop.
#950 mentions this, but was closed because swarm.key is not supported.

Additionally, the "this feature is not supported" error message showed in that first issue seems to be broken:

Describe the solution you'd like
I would like IPFS Desktop to allow users to connect to a private IPFS network using the configured swarm.key file stored at $IPFS_PATH/swarm.key (following the instructions here: https://github.com/ahester57/ipfs-private-swarm#2-generate-swarmkey)

Describe alternatives you've considered
The alternative is to use kubo CLI to run a private network, which is not viable becuase it is not user friendly for non technical users of IPFS.

Additional context
I work on the Social Impact team at the Filecoin Foundation, and some of our projects are looking to run private archival networks using IPFS. While the swarm key approach will allow technical users to manage a private network that replicates data, for smaller community archives and art galleries without strong in house technical support, being able to access the data on this private network via a friendly user interface is a critical requirement.

[lidel]

The swarm.key-based private network (aka PNET) is experimental in Kubo, but stable and as long users are okay with double-encryption and being limited to TCP-only, it could be supported.

It would require changes to ipfs-webui, to make it very, very clear from the GUI the node is running in private swarm mode and the feature set is severely limited.

@ianconsolata it sounds simple (just add file picket for swarm.key) but there is a lot of security risks around user-friendly app like this. We don't have Tor for a similar reason: we don't want to give users a false sense of security, nor a canon for less technical or security conscious users to shoot themselves in the foot.

It is difficult to tell how much work this feature request implies, mind elaborating on how end user's experience is expected to work? User has swarm.key, and selects it via UI. What happens next?

Some questions form the top of my head:

• We will be leaking metadata (user IP and the CID they try to fetch from private swarm) to IPNI at https://cid.contact, and results will not be useful;
  • Do we disable IPNI when PNet is set? I imagine yes?
• Public bootstrappers won't work with swarm.key present.
  • Do we detect there are no peers when swarm.key is set, and ask user to provide multiaddr to bootstrap with?
• What happens when user removes swarm.key?
  • What if user onl ywanted to test this feature, and wants to go back to public swarm?
  • Should there be UI warning user that entire repository is now exposed to the public swarm? Do we give option to export everything to a CAR and then purge the blockstore to avoid data exfiltration?

[ianconsolata]

Good questions! Let me start off with some general background, and then I can go into more specifics.

Background

The primary project I am working with right now is HistoryPin, run by Shift Collective. They are looking to create a prototype decentralized storage backend to support their community archives based on the product requirements and user research they performed last year.

They are mostly interested in using private networks to create redundant archives of potentially sensitive community data, however, deletion is one of their core requirements. So instead of using the public network we are aiming to create a network of private nodes run by trusted institutions (i.e. universities, museums, libraries, etc) that will respond to deletion requests in a timely manner. It would be nice if we could also facilitate access to this network by the less technical community archives themselves (not just by the institutional network of node providers), which is where IPFS Desktop comes in.

Questions

It is difficult to tell how much work this feature request implies, mind elaborating on how end user's experience is expected to work? User has swarm.key, and selects it via UI. What happens next?

So I think they might have to configure more than just the swarm key, probably at least a bootstrap node, and maybe some additional settings (like the IPNI stuff mentioned below). My expectation of the user journey is that they would download IPFS Desktop, edit some configuration settings, and then connect to the private network associated with that swarm key. From there, they would be able to do all the normal IPFS stuff (i.e. fetch files by CID, pin files locally to store a redundant copy, and add some files from their local filesystem to provide to the private network. I expect one of the larger institutions involved would be responsible for running the bootstrap node(s) for this private network. Search and discovery of CIDs would happy through the HistoryPin site, though perhaps IPNS could also be used to publish an up to date list of the files in each community archive to the private swarm.

We will be leaking metadata (user IP and the CID they try to fetch from private swarm) to IPNI at https://cid.contact/, and results will not be useful;
Do we disable IPNI when PNet is set?

I think disabling IPNI is the right approach for this use case, as folks would not care about fetching data from the public network. Leaking metadata about what folks are searching for would not be acceptable (for example, one of the archives is an LGBTQ archive in the south and those requesting data could be targeted if that information was leaked). Doxxing is one of the primary concerns for these community archives, and why they want a private network where data can easily be deleted.

Public bootstrappers won't work with swarm.key present.
Do we detect there are no peers when swarm.key is set, and ask user to provide multiaddr to bootstrap with?

Yes, please! Our plan is to have one of the institutional networks run a bootstrap node, and we would distribute that along with the swarm key to community archives.

What happens when user removes swarm.key?
What if user only wanted to test this feature, and wants to go back to public swarm?
Should there be UI warning user that entire repository is now exposed to the public swarm? Do we give option to export everything to a CAR and then purge the blockstore to avoid data exfiltration?

The data should definitely not get immediately published or provided to the public network if the swarm key is disabled.
I think maybe a separate blockstore per swarm key? It would be nice if users could toggle back and forth between networks, but that is very much a nice to have feature not a requirement. If that is infeasible, exporting the data and purging the blockstore would be the next best approach.

[lidel]

Ok, there are things which are time-consuming to implement, and some which we can cheese a little by reusing existing abstractions, such as the fact everything in Kubo (swarm keys, peer identity, configuration, data) is stored in a swappable .ipfs repository.

With that in mind, I think a realistic MVP would be based on three things:

• Ability to enable "private swarm" feature via tray menu
  • first step, file picker for selecting swarm.key from disk + enter "pet name" for the private swarm
    • We don't implement "generate swarm key" – for now this would be done by someone technical, who also sets up a private bootstrapper
  • second step, asking user to provide optional bootstrapper multiaddr (or option to continue without bootstrapper)
    • I assume LAN-only private swarms based on MDNS works, so bootstrapper would be optional, if not, then we can make this step mandatory.
• Explicitly isolate pnet repository at filesystem level
  • before enabling pnet, move old .ipfs to .ipfs.public
    • this ensures we dont mix public and private data, and allow user to switch back and forth
  • generate new .ipfs.private-swarm-{id} where {id} is based on swarm.key (e.g. 7 first characters of sha2 etc, or just the pet name user provided)
  • tray menu to allow user to switch between private swarm(s) and public one
    • I suspect supporting multiple private swarms implemented and isolated this way will be very inexpensive to support
• Ability for ipfs-webui to detect and show "private swarm mode"
  • Some indicator in the left side menu, always visible (e.g. "Private IPFS swarm {pet name}" under IPFS cube)
    • How to signal?
      • localStorage key could be set by Electron app (doable, no need for change in Kubo)
      • or /api/v0/id could have additional PNet key info field present (trivial, but requires kubo release)
  • Disable features that depend on public swarm, such as remote pinning, IPLD examples (which would not work anyway) etc.

It is bare-bones, but could be delivered (after IPFS Camp, late July or August) without sinking too much dev time, and not blocked by external dependencies.

@ianconsolata Thoughts? Would this MVP be enough?

[ianconsolata]

That would be a fantastic MVP, and I think would support our needs for now. Let me share this issue with a couple of the partners we are working with and let them weigh in with their thoughts, just in case there are other needs I have forgotten about.

[edsu]

This sounds amazing, and would be a huge step forward for community archives that want to get started with collaborative file sharing using IPFS.

It sounds like having multiple private swarms could allow community archives to participate in more than one trust network? We don't currently have any use cases for this at the moment, but I think it could be super interesting for us to explore.

[lidel]

The MVP hard limitation of 'only one swarm at a time' is what makes its implementation with preexisting Kubo feasible, but yes, in the future we could explore multi-swarm use case, if there is enough need. But that would be way bigger scope than MVP.

[ianconsolata]

Connecting to multiple swarms at once is interesting, but I still think it would be safer from a UX perspective to have folks actively switch from one swarm to another, rather than allowing both swarms to operate simultaneously. Having someone close IPFS Desktop (or shut down the ipfs daemon) and relaunch it with a different config gives clear separation between public and private networks with little chance someone would accidentally share or re-provide data to the wrong network.

I like that this MVP approach still allows for fast switching between networks very easily, since the block store and other settings stored in .ipfs get copied for each new network configured. It would result in some duplication if the same data is being provided to multiple networks, but I think that is an acceptable tradeoff for now. So this solution does effectively let folks participate in multiple networks, but only one at a time.