Skip to content

Commit

Permalink
Incorporate recent updates to halo2 branch (#1656)
Browse files Browse the repository at this point in the history
* fix: use current process binding to limit thread cores (#1633)

Use the current processes bound cores to limit the possible cores that
threads can be bound to. This allows core binding to work properly when the
lotus-worker service is limited to certain CPUs by cgroups.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* fix: broken Links in README.md #1637 (#1649)

* feat: Introduce PoRepConfig::new_groth16() (#1635)

Instead of constructing the `PoRepConfig` directly, use a constructor
function. This simplifies the code and makes things less error-prone.

Fixes #1632.

Co-authored-by: 卜翰 <[email protected]>

* fix: clean up tree definitions (#1655)

There were quite a few public type definitions, that were mostly replaced by
the `SectorShape*` types. This commit cleans them up and moves them around if
appropriate.

This should make the code easier to follow and the public API surface smaller.

BREAKING CHANGE: `BinaryLCMerkleTree`, `BinaryMerkleTree`,
`BinarySubMerkleTree`, `LCMerkleTree`, `LCStore`, `MerkleStore`, `MerkleTree`,
`OctLCMerkleTree`, `OctLCSubMerkleTree`, `OctLCTopMerkleTree`, `OctMerkleTree`,
`OctSubMerkleTree`, `OctTopMerkleTree`, `QuadLCMerkleTree` and `QuadMerkleTree`
are removed from the public interface.

* fix: update ec-gpu-gen (#1638)

`ec-gpu-gen` needs to be updated to v0.5 as v0.4 has dependencies that
depend on yanked version. It's an indirect dependency of `bellperson` and
`neptune`, which are upgraded here.

Moving from `memmap` (which is deprecated) to `memmap2` was also needed
als dependencies also switched.

`chrono` updated their API, so there was also a small change needed.

* chore: update Cargo.lock

* fix: there was a memmap -> memmap2 missing

* fix: make poseidon tests pass

Neptune currently is a fork of pasta_curves. That needs patching as
well, in order to get the correct names for the fields out.

* ci: Apply rustfmt and fix clippy

* Pick relevant branch of neptune

* fix: Use SHA256 hasher for binary trees

Co-authored-by: Clint Armstrong <[email protected]>
Co-authored-by: Volker Mische <[email protected]>
Co-authored-by: hanbu97 <[email protected]>
Co-authored-by: 卜翰 <[email protected]>
  • Loading branch information
5 people committed Jan 19, 2023
1 parent 5764f4b commit e0fdb29
Show file tree
Hide file tree
Showing 45 changed files with 677 additions and 696 deletions.
757 changes: 364 additions & 393 deletions Cargo.lock

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,7 @@ members = [
]

[patch.crates-io]
halo2_proofs = { git = "https://github.com/filecoin-project/halo2", branch = "serialize-vk-pk" }
halo2_proofs = { git = "https://github.com/filecoin-project/halo2", branch = "halo2" }
neptune = { git = "https://github.com/filecoin-project/neptune", branch = "halo2" }
pasta_curves = { git = "https://github.com/filecoin-project/pasta_curves", branch = "gpu" }
fil_pasta_curves = { git = "https://github.com/filecoin-project/pasta_curves", branch = "gpu-fil-fork" }
11 changes: 4 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,18 @@ The **Filecoin Proving Subsystem** (or FPS) provides the storage proofs required

There are currently several different crates:

- [**Storage Proofs (`storage-proofs`)**](./storage-proofs)
A library for constructing storage proofs – including non-circuit proofs, corresponding SNARK circuits, and a method of combining them.

- [**Storage Proofs Core (`storage-proofs-core`)**](./storage-proofs/core)
- [**Storage Proofs Core (`storage-proofs-core`)**](./storage-proofs-core)
A set of common primitives used throughout the other storage-proofs sub-crates, including crypto, merkle tree, hashing and gadget interfaces.

- [**Storage Proofs PoRep (`storage-proofs-porep`)**](./storage-proofs/porep)
- [**Storage Proofs PoRep (`storage-proofs-porep`)**](./storage-proofs-porep)
`storage-proofs-porep` is intended to serve as a reference implementation for _**Proof-of-Replication**_ (**PoRep**), while also performing the heavy lifting for `filecoin-proofs`.

Primary Components:
- **PoR** (**_Proof-of-Retrievability_**: Merkle inclusion proof)
- **DrgPoRep** (_Depth Robust Graph_ **_Proof-of-Replication_**)
- **StackedDrgPoRep**

- [**Storage Proofs PoSt (`storage-proofs-post`)**](./storage-proofs/post)
- [**Storage Proofs PoSt (`storage-proofs-post`)**](./storage-proofs-post)
`storage-proofs-post` is intended to serve as a reference implementation for _**Proof-of-Space-time**_ (**PoSt**), for `filecoin-proofs`.

Primary Components:
Expand All @@ -30,7 +27,7 @@ There are currently several different crates:

## Security Audits

The `rust-fil-proofs` proofs code and the [Filecoin Spec](https://bafybeidxw5vxjdwsun2zc2illagf43v6w5r5w63vg455h7vjesbyqssg64.ipfs.dweb.link/algorithms/sdr/) has undergone a [proofs security audit](audits/Sigma-Prime-Protocol-Labs-Filecoin-Proofs-Security-Review-v2.1.pdf) performed by [Sigma Prime](https://sigmaprime.io/) and been deemed free of *critical* or *major* security issues. In addition to the security review, the document provides the summary of findings, vulnerability classifications, and recommended resolutions. All known issues have been resolved to date in both the code and the specification.
The `rust-fil-proofs` proofs code and the [Filecoin Spec](https://spec.filecoin.io/algorithms/sdr/) has undergone a [proofs security audit](audits/Sigma-Prime-Protocol-Labs-Filecoin-Proofs-Security-Review-v2.1.pdf) performed by [Sigma Prime](https://sigmaprime.io/) and been deemed free of *critical* or *major* security issues. In addition to the security review, the document provides the summary of findings, vulnerability classifications, and recommended resolutions. All known issues have been resolved to date in both the code and the specification.

`rust-fil-proofs` has also undergone a [SNARK proofs security audit performed by Dr. Jean-Philippe Aumasson and Antony Vennard](audits/protocolai-audit-20200728.pdf) and been deemed free of *critical* or *major* security issues. In addition to the security analysis, the document provides the audit goals, methodology, functionality descriptions and finally observations on what could be improved. All known issues have been resolved to date.

Expand Down
2 changes: 1 addition & 1 deletion fil-halo2-gadgets/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ repository = "https://github.com/filecoin-project/rust-fil-proofs"
ff = "0.12.0"
halo2_proofs = "0.2.0"
halo2_gadgets = "0.2.0"
neptune = { git = "https://github.com/filecoin-project/neptune", branch = "halo2" }
neptune = "~8.1.0"
sha2 = "0.9.2"

[dev-dependencies]
Expand Down
4 changes: 2 additions & 2 deletions fil-halo2-gadgets/src/bin/sha256_compute_k.rs
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ fn sha256_chip_compute_k() {
Ok(_) => {
// Proof size does not change with circuit size.
if num_blocks == PREIMAGE_BLOCK_SIZES.start {
let cost = CircuitCost::<Eq, MyCircuit>::measure(k as usize, &circ);
let cost = CircuitCost::<Eq, MyCircuit>::measure(k, &circ);
let proof_size: usize = cost.marginal_proof_size().into();
println!("\tproof_size = {} bytes", proof_size);
}
Expand Down Expand Up @@ -219,7 +219,7 @@ fn compress_chip_compute_k() {
Ok(_) => {
// Proof size does not change with circuit size.
if num_blocks == PREIMAGE_BLOCK_SIZES.start {
let cost = CircuitCost::<Eq, MyCircuit>::measure(k as usize, &circ);
let cost = CircuitCost::<Eq, MyCircuit>::measure(k, &circ);
let proof_size: usize = cost.marginal_proof_size().into();
println!("\tproof_size = {} bytes", proof_size);
}
Expand Down
2 changes: 1 addition & 1 deletion fil-proofs-param/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ structopt = "0.3.12"
humansize = "1.1.0"
indicatif = "0.16.2"
dialoguer = "0.10.0"
blstrs = "0.5.0"
blstrs = "0.6.1"
typenum = "1.11.2"
generic-array = "0.14.4"
halo2_proofs = "0.2.0"
Expand Down
28 changes: 3 additions & 25 deletions fil-proofs-param/src/bin/paramcache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use blstrs::Scalar as Fr;
use dialoguer::{theme::ColorfulTheme, MultiSelect};
use filecoin_proofs::{
constants::{
DefaultPieceHasher, POREP_PARTITIONS, PUBLISHED_SECTOR_SIZES, WINDOW_POST_CHALLENGE_COUNT,
DefaultPieceHasher, PUBLISHED_SECTOR_SIZES, WINDOW_POST_CHALLENGE_COUNT,
WINDOW_POST_SECTOR_COUNT, WINNING_POST_CHALLENGE_COUNT, WINNING_POST_SECTOR_COUNT,
},
parameters::{public_params, window_post_public_params, winning_post_public_params},
Expand Down Expand Up @@ -226,18 +226,7 @@ fn generate_params_porep(sector_size: u64, api_version: ApiVersion) {
sector_size,
Fr,
cache_porep_params,
PoRepConfig {
sector_size: SectorSize(sector_size),
partitions: PoRepProofPartitions(
*POREP_PARTITIONS
.read()
.expect("POREP_PARTITIONS poisoned")
.get(&sector_size)
.expect("unknown sector size") as usize,
),
porep_id: [0; 32],
api_version,
}
PoRepConfig::new_groth16(sector_size, [0; 32], api_version)
);
}

Expand All @@ -246,18 +235,7 @@ fn generate_params_empty_sector_update(sector_size: u64, api_version: ApiVersion
sector_size,
Fr,
cache_empty_sector_update_params,
PoRepConfig {
sector_size: SectorSize(sector_size),
partitions: PoRepProofPartitions(
*POREP_PARTITIONS
.read()
.expect("POREP_PARTITIONS poisoned")
.get(&sector_size)
.expect("unknown sector size") as usize,
),
porep_id: [0; 32],
api_version,
}
PoRepConfig::new_groth16(sector_size, [0; 32], api_version)
);
}

Expand Down
6 changes: 3 additions & 3 deletions fil-proofs-tooling/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ serde_json = "1.0"
regex = "1.3.7"
commandspec = "0.12.2"
chrono = { version = "0.4.7", features = ["serde"] }
memmap = "0.7.0"
bellperson = "0.22.0"
memmap2 = "0.5.6"
bellperson = "0.24.0"
rand = "0.8"
tempfile = "3.0.8"
cpu-time = "1.0.0"
Expand All @@ -47,7 +47,7 @@ fdlimit = "0.2.0"
dialoguer = "0.10.0"
structopt = "0.3.12"
humansize = "1.1.0"
blstrs = "0.5.0"
blstrs = "0.6.1"
time = "0.3.9"
pasta_curves = "0.4.0"

Expand Down
5 changes: 2 additions & 3 deletions fil-proofs-tooling/src/bin/benchy/prodbench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -170,9 +170,8 @@ fn configure_global_config(inputs: &ProdbenchInputs) {
.expect("POREP_PARTITIONS poisoned")
.insert(inputs.sector_size_bytes(), inputs.porep_partitions);
POREP_MINIMUM_CHALLENGES
.write()
.expect("POREP_MINIMUM_CHALLENGES poisoned")
.insert(inputs.sector_size_bytes(), inputs.porep_challenges);
.get_mut()
.insert(inputs.sector_size_bytes(), inputs.porep_challenges as usize);
}

pub fn run<F>(
Expand Down
7 changes: 1 addition & 6 deletions fil-proofs-tooling/src/bin/circuitinfo/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -159,12 +159,7 @@ fn porep_info(sector_size: u64, api_version: ApiVersion) -> (CircuitInfo, usize)
sector_size,
Fr,
get_porep_info,
PoRepConfig {
sector_size: SectorSize(sector_size),
partitions,
porep_id: [0; 32],
api_version,
}
PoRepConfig::new_groth16(sector_size, [0; 32], api_version)
);
(info, partitions.into())
}
Expand Down
11 changes: 5 additions & 6 deletions fil-proofs-tooling/src/bin/update_tree_r_cache/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,16 @@ use filecoin_proofs::{
SectorShapeSub2, SectorShapeSub8, SectorShapeTop2, OCT_ARITY,
};
use generic_array::typenum::Unsigned;
use memmap::MmapOptions;
use memmap2::MmapOptions;
use merkletree::{
merkle::get_merkle_tree_len,
store::{ExternalReader, ReplicaConfig, Store, StoreConfig},
store::{ExternalReader, LevelCacheStore, ReplicaConfig, Store, StoreConfig},
};
use pasta_curves::{Fp, Fq};
use storage_proofs_core::{
cache_key::CacheKey,
merkle::{
create_lc_tree, get_base_tree_count, split_config_and_replica, LCStore, LCTree,
MerkleTreeTrait,
create_lc_tree, get_base_tree_count, split_config_and_replica, LCTree, MerkleTreeTrait,
},
util::{default_rows_to_discard, NODE_SIZE},
};
Expand Down Expand Up @@ -75,7 +74,7 @@ where
let base_tree_len = get_merkle_tree_len(base_tree_leafs, OCT_ARITY)?;
let tree_r_last_root = if is_sector_shape_base(sector_size) {
ensure!(configs.len() == 1, "Invalid tree-shape specified");
let store = LCStore::<DefaultTreeDomain<F>>::new_from_disk_with_reader(
let store = LevelCacheStore::new_from_disk_with_reader(
base_tree_len,
OCT_ARITY,
&configs[0],
Expand Down Expand Up @@ -255,7 +254,7 @@ where
// First, read the roots from the cached trees on disk
let mut cached_base_tree_roots: Vec<DefaultTreeDomain<F>> = Vec::with_capacity(tree_count);
for (i, config) in configs.iter().enumerate().take(tree_count) {
let store = LCStore::new_from_disk_with_reader(
let store = LevelCacheStore::new_from_disk_with_reader(
base_tree_len,
OCT_ARITY,
config,
Expand Down
2 changes: 2 additions & 0 deletions fil-proofs-tooling/src/metadata.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ impl GitMetadata {
let repo = Repository::discover(&repo_path)?;
let head = repo.head()?;
let commit = head.peel_to_commit()?;

// Unwrap is OK as the given seconds won't be out of range.
let date = Utc.timestamp_opt(commit.time().seconds(), 0).unwrap();

Ok(GitMetadata {
Expand Down
34 changes: 6 additions & 28 deletions fil-proofs-tooling/src/shared.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ use filecoin_hashers::Hasher;
use filecoin_proofs::{
add_piece, fauxrep_aux, seal_pre_commit_phase1, seal_pre_commit_phase2,
validate_cache_for_precommit_phase2, DefaultPieceHasher, DefaultTreeHasher, MerkleTreeTrait,
PaddedBytesAmount, PieceInfo, PoRepConfig, PoRepProofPartitions, PoStConfig, PoStType,
PrivateReplicaInfo, PublicReplicaInfo, SealPreCommitOutput, SealPreCommitPhase1Output,
SectorSize, UnpaddedBytesAmount, POREP_PARTITIONS, WINDOW_POST_CHALLENGE_COUNT,
WINDOW_POST_SECTOR_COUNT, WINNING_POST_CHALLENGE_COUNT, WINNING_POST_SECTOR_COUNT,
PaddedBytesAmount, PieceInfo, PoRepConfig, PoStConfig, PoStType, PrivateReplicaInfo,
PublicReplicaInfo, SealPreCommitOutput, SealPreCommitPhase1Output, SectorSize,
UnpaddedBytesAmount, WINDOW_POST_CHALLENGE_COUNT, WINDOW_POST_SECTOR_COUNT,
WINNING_POST_CHALLENGE_COUNT, WINNING_POST_SECTOR_COUNT,
};
use generic_array::typenum::Unsigned;
use log::info;
Expand Down Expand Up @@ -137,18 +137,7 @@ where
let sector_size_unpadded_bytes_ammount =
UnpaddedBytesAmount::from(PaddedBytesAmount::from(sector_size));

let porep_config = PoRepConfig {
sector_size,
partitions: PoRepProofPartitions(
*POREP_PARTITIONS
.read()
.expect("poisoned read access")
.get(&u64::from(sector_size))
.expect("unknown sector size") as usize,
),
porep_id,
api_version,
};
let porep_config = PoRepConfig::new_groth16(u64::from(sector_size), porep_id, api_version);

let mut out: Vec<(SectorId, PreCommitReplicaOutput<Tree>)> = Default::default();
let mut sector_ids = Vec::new();
Expand Down Expand Up @@ -343,18 +332,7 @@ pub fn get_porep_config<F: PrimeField>(sector_size: u64, api_version: ApiVersion
let arbitrary_porep_id = [99; 32];

if util::is_groth16_field::<F>() {
PoRepConfig {
sector_size: sector_size.into(),
partitions: PoRepProofPartitions(
*POREP_PARTITIONS
.read()
.expect("POREP_PARTITONS poisoned")
.get(&(sector_size))
.expect("unknown sector size") as usize,
),
porep_id: arbitrary_porep_id,
api_version,
}
PoRepConfig::new_groth16(sector_size, arbitrary_porep_id, api_version)
} else {
PoRepConfig::new_halo2(sector_size.into(), arbitrary_porep_id, api_version)
}
Expand Down
13 changes: 6 additions & 7 deletions filecoin-hashers/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,16 @@ repository = "https://github.com/filecoin-project/rust-fil-proofs"
readme = "README.md"

[dependencies]
bellperson = "0.22.0"
blstrs = "0.5.0"
bellperson = "0.24.0"
blstrs = "0.6.1"
generic-array = "0.14.4"
merkletree = "0.22.0"
ec-gpu = "0.1.0"
ec-gpu = "0.2.0"
ff = "0.12.0"
anyhow = "1.0.34"
serde = "1.0.117"
rand = "0.8.0"
# neptune = { version = "~7.0.0", optional = true, features = ["bls", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
neptune = { git = "https://github.com/filecoin-project/neptune", branch = "halo2", optional = true, features = ["bls", "pasta", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
neptune = { version = "~8.1.0", optional = true, features = ["bls", "pasta", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
lazy_static = { version = "1.4.0", optional = true }
blake2s_simd = { version = "1.0.0", optional = true }
sha2 = { version = "0.10.2", optional = true }
Expand All @@ -31,8 +30,8 @@ fil-halo2-gadgets = { path = "../fil-halo2-gadgets" }
[features]
default = ["opencl", "blake2s", "poseidon", "sha256"]

cuda = ["bellperson/cuda", "neptune/cuda"]
opencl = ["bellperson/opencl", "neptune/opencl"]
cuda = ["bellperson/cuda", "halo2_proofs/cuda", "neptune/cuda"]
opencl = ["bellperson/opencl", "halo2_proofs/opencl", "neptune/opencl"]

# available hashers
blake2s = ["blake2s_simd"]
Expand Down
9 changes: 6 additions & 3 deletions filecoin-hashers/src/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ use bellperson::{
ConstraintSystem, SynthesisError,
};
use blstrs::Scalar as Fr;
use ec_gpu::GpuField;
use ff::{Field, PrimeField};
use fil_halo2_gadgets::ColumnCount;
use halo2_proofs::{
Expand Down Expand Up @@ -44,7 +43,7 @@ pub trait Domain:
+ Element
+ StdHash
{
type Field: PrimeField + GpuField;
type Field: PrimeField;

#[allow(clippy::wrong_self_convention)]
fn into_bytes(&self) -> Vec<u8> {
Expand Down Expand Up @@ -99,7 +98,11 @@ pub trait HashFunction<T: Domain>: Clone + Debug + Send + Sync + LightAlgorithm<
}

pub trait Hasher: Clone + Debug + Eq + Default + Send + Sync {
type Field: PrimeField + GpuField;
#[cfg(not(any(feature = "cuda", feature = "opencl")))]
type Field: PrimeField;
#[cfg(any(feature = "cuda", feature = "opencl"))]
type Field: PrimeField + ec_gpu::GpuName;

type Domain: Domain<Field = Self::Field> + LightHashable<Self::Function> + AsRef<Self::Domain>;
type Function: HashFunction<Self::Domain>;

Expand Down
6 changes: 3 additions & 3 deletions filecoin-proofs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ storage-proofs-update = { path = "../storage-proofs-update", version = "~12.0.0"
filecoin-hashers = { version = "~7.0.0", path = "../filecoin-hashers", default-features = false, features = ["poseidon", "sha256"] }
rand = "0.8"
lazy_static = "1.2"
memmap = "0.7"
memmap2 = "0.5.6"
serde = { version = "1.0", features = ["rc", "derive"] }
serde_json = "1.0"
ff = "0.12.0"
blake2b_simd = "1.0.0"
bellperson = "0.22.0"
bellperson = "0.24.0"
log = "0.4.7"
rayon = "1.1.0"
hex = "0.4.0"
Expand All @@ -34,7 +34,7 @@ gperftools = { version = "0.2", optional = true }
generic-array = "0.14.4"
fr32 = { path = "../fr32", version = "~5.0.0", default-features = false }
once_cell = "1.8.0"
blstrs = "0.5.0"
blstrs = "0.6.1"
halo2_proofs = "0.2.0"

[dev-dependencies]
Expand Down
Loading

0 comments on commit e0fdb29

Please sign in to comment.