Skip to content

Latest commit

 

History

History
184 lines (144 loc) · 14.9 KB

0. Compile and deploy DIY OOD.md

File metadata and controls

184 lines (144 loc) · 14.9 KB
Raw

Introduction

Thank you very much for reading the Hello, CYFS tutorial series!

Based on what we've learned about engineers over the years, this should be the first article you read after you decide to get "serious" about using CYFS. And, you expect nothing nebulous in this series, just practical, hands-on content.

Yes, in the Hello, CYFS tutorial series, we do not require you to have read other CYFS-related articles, nor do we require you to have extensive background knowledge of the Web3 industry. We only assume that you already have the following knowledge:

  1. be relatively proficient in using the Linux operating system
  2. can JavaScript, can read and understand TypeScript (we strongly recommend that every JavaScript developer learn to use TypeScript)
  3. familiar with common JS scaffolding
  4. have a basic understanding of the HTTP protocol

This tutorial will guide you to create a standard CYFS Object URL (cyfs://o/$owner_id/$obj_id), as fast as possible to understand how CYFS implements the Web3 concept of "ownership", and how Web3 “Ownership means that everyone has the ability to save content, publish content, and earn direct revenue from content on the Web”.

What is CYFS Object URL

The HTTP protocol widely used today has a simple design: http://$host/$path , which logically means that it points to a file stored at $path on the computer file system represented by $host. This design has several well-known drawbacks:

  1. 404 problems, whether it is a computer network failure which $host points to, or the adjustment of its storage structure, can cause 404.
  2. The inability to find a large number of different URLs pointing to the same content.
  3. URL generation is posterior, you must have the domain name and server frist, and then generate it after the completion of deployment.
  4. no content validation, the owner of $host can modify the content of the URL at will.
  5. no content validation, can only use the source trusted: trust all content returned by the server holding the certificate.

Under the influence of these shortcomings, webmasters increasingly dislike using external links ($host is someone else's URL) when building content in order to improve the visiting experience of their websites, and eventually, the Web that links everything is curled up into an information silo.

Solving these problems and making information interconnected is the first task of the Web3 URL represented by cyfs://. We use "Content base URL" to solve this problem. A typical CYFS URL is defined as follows.

cyfs://o/$owner_id/$obj_id

The o in the domain segment indicates that this is a CYFS Object Link, and the Object Link points to a definite content (which we call a NamedObject). The $owner_id represents the address of the owner of this content, which can simply be interpreted as the Hash of the public key, and the $obj_id can simply be interpreted as the Hash of the content. this means that a legitimate CYFS Object Link can be constructed as long as the content and the author of the content are identified. this is a clean, topology-independent URL construct. It has the following advantages over HTTP URLs.

  1. topology-independent, allowing URLs to be constructed offline
  2. the content returned from any server is valid as long as the hash is correct
  3. the content is verifiable, so don’t worry about tampering

Parsing of CYFS Object URLs

After understanding the basic principle of CYFS Object URL, we know that it is simple to construct CYFS Object URL for any file by cryptography, but how is this URL opened? Instead of using a slow DHT for decentralized content addressing, we introduce a new node called OOD (Owner Onine Device) in the network topology to simply solve this problem. The routine steps are as follows: Step1: Get the OOD’s DeviceId by $owner_id registered on the blockchain Step2: Establish a P2P link to OOD by DeviceId, requesting $obj_id Step3: DeviceId returns the content corresponding to $obj_id

It seems to be not much different from the standard HTTP protocol resolution process, but here we avoid the use of centralized domain name system and CA system. We also give the system a great automatic load balancing capability by introducing NDN theory in Step3. But this is just a quick primer, so we won't expand on it.

After understanding the above steps, we can think that there are 3 steps to create a CYFS Object URL that can be accessed.

  1. create the cryptographic identity, which we usually call the CreatePeopleObject
  2. create the OOD and bind the OOD to the cryptographic identity on the blockchain
  3. calculate the hash of the file and upload the file to the OOD

Build DIY OOD

Let's start with something relatively slow: prepare the OOD. ood can be thought of, in technical terms, as a personal compute node running a specific service with relatively good network access (able to be successfully linked by P2P). Let's prepare the compute node first. We recommend a Linux VPS (with 4G RAM) or an idle Linux host, which preferably has a good network environment (IPv6 or IPv4 with good NAT).

Deploy the relevant services on the prepared compute node to be called an OOD, with the option of:

  1. install using our pre-compiled binary installer
  2. compile your own OOD Service components from source and deploy them yourself

Use the pre-compiled binary installer

CYFS will have three different version lines, which are isolated from each other by default. CYFS nodes that use different version lines do not interoperate with each other. We do not guarantee data compatibility between the three version lines

  • Nightly: The Nightly version is the most frequently updated version, and all new features and modifications will be reflected first in the Nightly version, which is used by the CYFS Core Dev Team for development.
  • Beta: A general sense test release that periodically merges features from the Nightly release. It is recommended for external developers to use this version for early testing and development, and to ensure compatibility and reliability of data and protocols between Beta versions.
  • Stable: Stable version. Features from the Beta version are periodically merged, and compatibility and reliability of data and protocols between Stable versions is ensured. Currently, CYFS has not released a stable version yet.

You can download it from [download.md](... /Download.md). /Download.md) or our official website https://cyfs.com/download, to download the corresponding system's OOD binary installation package. After downloading, just execute the installation package directly. This installer will do the following.

  • If it is a non-windows system, check the libc version to make sure it meets our pre-compiled binary requirements
  • If the system does not have node.js installed, install the LTS version of node.js. The installer does not check for an installed version of node.js
  • If the system is non-windows and docker is not installed, install the latest version of docker-ce
  • Download the latest OOD Service components from the official Service Repo server of the current version line (nightly versions are updated very frequently)
  • Deploy OOD Service locally and start

The pre-compiled binary installer has been tested on the following systems.

  • Windows 10/11
  • Ubuntu 18/20
  • Centos 7/8
  • Debian 9

Other Windows, Ubuntu, Debian, RedHat not listed may also be supported. We have not tested this completely on more systems.

The following directories were added to the system after the installation was completed. Some directories may not appear just after the installation is complete, here are just as examples.

{cyfs_root}
    ├── app
    │ └── web
    ├── data
    ├── etc
    │ ├── acl
    │ ├── desc
    │ ├── gateway
    │ └── ood-daemon
    ├─ log
    │ ├── app
    │ ├── ood-daemon
    │ ├── ood-daemon-monitor
    │ └─ panic
    ├─ run
    ├─ services
    │ ├── app-manager
    │ ├── chunk-manager
    │ ├── dsg-service
    │ ├── file-manager
    │ ├─ gateway
    │ └── ood-daemon
    └─ tmp
        └─ repo
  • {cyfs_root}: The root directory of the CYFS OOD component. all data related to OOD will be placed in this directory. The exact location of the directory is.
    • ${cyfs_root} specific path.
    • Windows: c:\cyfs
    • MacOS: ~/Library/cyfs
    • Other systems: /cyfs
  • app: The program where Dec App is stored. dec Service part is stored under {DEC_ID} subfolder, Dec Web part is stored under web\{DEC_ID} subfolder.
  • data: Local data storage directory, OOD's various services and Dec Service's local data will be stored in this folder.
  • etc: Configuration storage directory. Various configuration files and OOD's identity files are stored in this directory.
  • log: log storage directory, OOD service logs are stored in {service_name} subdirectory, Dec Service logs are generally stored in app\{app_name} subdirectory.
  • services: The program where the OOD service is stored. We can see that there are 6 OOD service programs, which are stored in the subfolder named service_name.
  • run: the directory where the process lock files are stored.
  • tmp: Temporary directory for temporary files generated by the OOD service.

After installation and before binding, there will be only one ood-daemon process running, waiting for the OOD to be bound. When OOD is bound, all 6 OOD service programs will be running. We will explain how to bind OOD in the next chapter.

Compile your own OOD Service components from source code and deploy

If your target system is not on the supported list, you can also choose to compile and deploy the OOD Service by yourself. CYFS OOD theoretically supports the vast majority of Windows and Linux/Unix-like systems If you are using a system that is not in the above list, for example, you plan to build OOD with your own Raspberry Pi, you will need to compile the code yourself at this time. Official binary support for Raspberry Pi is in preparation If you don't want to use pre-compiled binaries, or if you make your own changes to the code, we also support compiling all the components needed for OOD yourself and deploying them to any system machine you install

Pre-compilation preparation

Currently, the compilation scripts we provide do not support cross-compilation, but you can adapt the scripts we provide to achieve cross-compilation.

The compilation of OOD Service needs to meet the following requirements.

  • Node.js 14 or above: required to execute the compilation script, this dependency is not required if the compilation script is not used. A large number of our compilation scripts are written in typescript, and in the CYFS ecosystem, rust and typescript are the languages for which we provide full support.
  • rustc version 1.57 or above Some compilation tools do not natively support the Apple M1/M2 platform yet.

Compiling OOD Service from scripts

The source code used by the different version lines can be obtained by checkingout different branches: > - nightly version line: maestro

  • nightly version line: main branch
  • beta version line: beta branch
  • stable version line: not ready yet

First, in the root directory of the source code (./src/), execute the command npm i to install the various dependencies needed by the script

The script needs to create the ${cyfs_root} folder during the compilation process. If the current system or the current user does not have permission to create this folder, you need to create the ${cyfs_root} folder manually and give read/write permission to the current user. (In a formal environment, all components and data of OOD System will be installed to ${cyfs_root})

Specific path to ${cyfs_root}.

  • Windows: c:\cyfs
  • MacOS: ~/Library/cyfs
  • Other systems: /cyfs

In the source code . /src/ folder, execute the command node ../scripts/build-standalone.js, this script will compile the OOD Service and the tools needed for the local deployment, and complete the initial local deployment. Finally the installer ood-installer is generated

Before compiling, the minor version number and the version line used by the Service can be specified via the VERSION and CHANNEL environment variables, with the optional value of CHANNEL currently being nightly or beta. If these two environment variables are not specified, the default minor version number is 0 and the version line is nightly.

Note: Regardless of which branch is checkedout, the version line must be specified via the CHANNEL variable before compilation. Mismatch between version line and branch may cause the compiled program to run incorrectly.

For details of the compilation steps, please refer to the build-standalone.js code.

Deploying OOD Service

When the compilation is done, it will

  1. generate ${cyfs_root}/repo_store directory, which stores the local compiled OOD Service binary files in a certain format. 2.
  2. generate the ood-installer installer in the src folder.

Before installation, you need to prepare the OOD dependency environment by yourself:

  • Node.js LTS version

If it is a non-Windows environment, you also need to:

  • Install the latest version of docker-ce and configure bootup
  • Execute the command docker pull alexsunxl/cyfs-base to pull the docker base image required to run Dec Service

Execute the command ./ood-installer --target solo, the locally compiled binary will be used as the OOD Service

The compiled OOD Service component can be used to deploy to other machines on the same system:

Copy the ${cyfs_root}/repo_store directory to the same directory on the new machine, copy the ood-installer program to any directory on the new machine, configure the dependency environment, and execute the above installation command. You can also build your own OOD Service Repo Server to deploy your own compiled OOD Service to a large number of machines in a single environment, which will be described in a more advanced document.

Other ways to build OOD

In addition to building your own DIY OOD, we also offer three more ways to get OOD for the average person. This is the most mainstream way in the future, and there are already several CYFS ecosystem companies that are customizing their own brand of OOD hardware on top of the CYFS open source system, which will soon be available in the market 2. Subscribe to the immediately available VirtualOOD based on VPS, VirtualOOD is the fastest way to open the lowest threshold OOD, I believe it will be the first choice of most ordinary users. There are some NAS enthusiasts in the CYFS ecosystem who are working on OOD kits for specific NAS models.

The use of several of the above OODs is intuitively guided by the Add OOD interface in CyberChat, the official CYFS wallet, so developers don't have to worry about end users not getting OODs.

Summary

By the end of this chapter, you will have your own OOD and a basic understanding of the CYFS open source code. In the next chapter, we will discuss how to have a valid cryptographic identity.