Skip to content

nickjiang2378/hydroflow

This branch is 2 commits ahead of, 116 commits behind hydro-project/hydroflow:main.

Folders and files

NameName
Last commit message
Last commit date
Jul 25, 2024
Feb 26, 2024
Aug 30, 2024
Aug 22, 2024
Aug 28, 2024
Aug 25, 2024
Jul 16, 2024
Aug 22, 2024
Sep 2, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 24, 2024
Aug 22, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 30, 2024
Aug 24, 2024
Aug 24, 2024
Aug 30, 2024
Aug 24, 2024
Nov 23, 2022
Jul 31, 2024
Apr 3, 2023
Jul 15, 2024
Aug 30, 2024
Aug 22, 2024
Dec 22, 2023
Oct 11, 2022
Jul 15, 2024
Aug 30, 2024
Jul 4, 2023
Aug 23, 2024
Jun 13, 2024
Jun 18, 2024
May 21, 2023

Repository files navigation

"hf" Hydroflow

Crates.io Docs.rs

Hydroflow is a low-latency dataflow runtime written in Rust. The goal of the Hydro Project is to empower developers to harness the full potential of the cloud by making distributed programs easy to specify and automatic to scale. Hydroflow is the lowest level in the Hydro stack, serving as a single-node low-latency runtime with explicit networking. This allows us to support not just data processing pipelines, but distributed protocols (e.g. Paxos) and real-world long-running applications as well.

Take a look at the Hydroflow Book.

The Hydroflow Surface Syntax

Hydroflow comes with a custom "surface syntax"—a domain-specific language which serves as a very simple, readable IR for specifying single-node Hydroflow programs. These programs are intended to be stitched together by the Hydro stack to create larger autoscaling distributed systems.

Here's a simple example of the surface syntax. Check out the Hydroflow Playground for an interactive demo.

source_iter(0..10)
  -> map(|n| n * n)
  -> filter(|n| *n > 10)
  -> foo;

foo = map(|n| (n..=n+1))
  -> flatten()
  -> for_each(|n| println!("Howdy {}", n));

For more, check out the surface syntax section of the Hydroflow book.

Start with a Template Program

We provide a cargo-generate template for you to get started from a simple working example.

To install cargo-generate, run the following:

cargo install cargo-generate

Then run:

cargo generate gh:hydro-project/hydroflow template/hydroflow

cd into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:

#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test

And you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice for adapting it to other uses.

Enable IDE Support for Ligatures

Since flow edges -> appear frequently in flows described using the Hydroflow surface syntax, enabling ligature support in your IDE may improve your code reading experience. This has no impact on code functionality or performance.

Instructions to enable this for the Fira Code font:

More font options are available here.

Dev Setup

See the setup section of the book.

The Examples Container

The hydroflow/examples subdirectory of this repository includes a number of examples. To make running these examples in the cloud easier, we've created a Docker image that contains compiled versions of those examples. The image is defined in the Dockerfile in the same directory as this README.

If you want to build the examples container locally, you can run

docker build -t hydroflow-examples .

This will build an image suitable for your architecture.

The scripts/multiplatform-docker-build.sh <image name> script will build both arm64 and amd64 versions of the image and push them to the image name specified. By default, this will push the image to DockerHub; if you want to push the image to another repository, you can pass an image URL as the argument to multiplatform-docker-build.sh instead.

Example binaries are located in /usr/src/myapp.

About

Hydro's low-level dataflow runtime

Resources

License

Citation

Stars

Watchers

Forks

Releases

No releases published

Languages

  • Rust 97.8%
  • Python 2.1%
  • Other 0.1%