Package Hermes #5

JonathanLorimer · 2021-07-21T16:32:05Z

Was able to get hermes to build and run due to a crate2nix branch that fixes some of the underlying issues I was running into.

Solves #4

JonathanLorimer · 2021-07-21T16:39:04Z

Github actions are failing because of a known bug with flakes and import from derivation NixOS/nix#4265 (comment)

JonathanLorimer · 2021-07-21T17:42:22Z

Okay, I think the best approach is to create a flake per project we want, and import them all into the top level flake!

JonathanLorimer · 2021-07-21T21:07:08Z

Okay, I switched to sub-flakes (which I think is nice, because then people can just point to that url if they want a subflake only). It still didn't resolve the import from derivation issue. This only breaks the commands nix flake show and nix flake check, which kinda sucks but I am gonna punt on this till later.

JonathanLorimer · 2021-07-21T22:36:38Z

SORTED OUT THE ISSUE WITH IFD. Its a bit of a hack, but should get things rolling for now, and as long as the flake is cached it shouldn't be a problem. Hopefully this gets fixed upstream.

soareschen · 2021-07-23T12:16:20Z

flake.lock

+        "type": "github"
+      }
+    },
+    "nixpkgs_4": {


Do you know why Nix Flake is generating many versions of the same package in the lock file?

soareschen · 2021-07-23T12:23:03Z

hermes/flake.nix

+          hermes =
+            let
+              name = "ibc-rs";
+              # The use of `evalPkgs` here is a major hack. This is to get around the IFD limitation in nix flakes


I am getting a warning:

trace: warning: crate2nix: Passing `buildRustCrate` as argument to Cargo.nix is deprecated. If you don't customize `buildRustCrate`, replace `callPackage ./Cargo.nix {}` by `import ./Cargo.nix { inherit pkgs; }`, and if you need to customize `buildRustCrate`, use `buildRustCrateForPkgs` instead.

Is it related to this workaround?

Ah yeah, that's because I need to use a fork of crate2nix. Right now crate2nix does not use fetchGit under the hood, so it only looks for the master branch. ibc-rs uses main I believe.

crate2nix = { url = "github:yusdacra/crate2nix/feat/builtinfetchgit"; flake = false; };

Here is the relevant issue upstream nix-community/crate2nix#205

I see. I thought the input for flakes is fixed, so inside a flake the Nix code shouldn't able to do fetchGit directly? Is the argument ibc-rs-src in the output closure just a URL, or is it a derivation containing the fetched source code already?

So, crate2nix isn't a flake. I know they keep track of git resource hashes using a json file.

Here is the relevant code: https://github.com/yusdacra/crate2nix/blob/master/tools.nix#L234-L241

soareschen

Works great on my machine!

soareschen · 2021-07-23T12:39:33Z

hermes/flake.nix

+        };
+
+      # nix build
+      defaultPackage = packages.hermes;


Would it be better to not provide a default package? I'd imagine as we add new projects, the user might not want to build or run hermes by default.

So I agree for the top level flake, gonna make that change. I think for the hermes/flake.nix the user probably just wants nix run to invoke the ibc-relayer-cli. I am interested to hear your thoughts here.

Right, I'm referring to the top level project. Got mixed up which file I'm looking at.

soareschen · 2021-07-23T12:39:35Z

hermes/flake.nix

+
+      # nix run
+      apps = {
+        hermes = {


Should we add documentation on README.md on what commands can be run? It is pretty tricky for new users to figure out what Nix commands to run. From here I can see two commands available: nix build .#hermes and nix run .#hermes.

Yeah, that's a good idea!

JonathanLorimer · 2021-07-28T15:53:13Z

@soareschen no rush on this, but its ready for re-review. Also sorry this has all taken so long. I didn't realize that nix + rust was so complicated (perhaps just bias since I am sure newcomers to cabal2nix would have the same complaint). FWIW I think the go projects will be easier to package (it makes me kinda sad to say that lol).

soareschen · 2021-08-03T09:25:02Z

When I first tried running the updated branch from the project root I got the error:

$ nix run .#hermes
error: NAR hash mismatch in input 'path:./hermes?narHash=sha256-Ug%2fOjuuCulhd79RCFDJLSMSPjcyvY1+XY6vwEFjaEPQ=' (/nix/store/z85653a74ggx4w9bzg8ppj
5q9m5dggff-source), expected 'sha256-Ug/OjuuCulhd79RCFDJLSMSPjcyvY1+XY6vwEFjaEPQ=', got 'sha256-Lnqh0y13bU6/7A4Rv7WM0aQzioLmkdJ8k3nh+V3CH8c='     
(use '--show-trace' to show detailed location information)

I had to go into the ./hermes directory and run nix run there first, then the error is gone after going back to the project root.

Could this be caused by local changes in the local ./hermes directory, which causes the checksum to change? Are gitignored files also tracked in the flake checksum?

JonathanLorimer · 2021-08-06T19:36:13Z

@soareschen I am having trouble reproducing locally, even after running nix store delete .#hermes. However, I did notice this same error occurring in the CI I setup in my personal repo. It seems strange that the hashes would be different, and I am having trouble tracking the source of nondeterminism. It could be from the use of fetchgit in crate2nix.

JonathanLorimer · 2021-08-06T19:58:22Z

Okay, I was able to reproduce by running these commands:

$> nix nar dump-path ./hermes > hermes.nar
$> nix hash-file --type sha256 ./hermes.nar
warning: 'hash-file' is a deprecated alias for 'hash file'
sha256-Lnqh0y13bU6/7A4Rv7WM0aQzioLmkdJ8k3nh+V3CH8c=
$> cd hermes
$> nix build
$> cd ..
$> nix nar dump-path ./hermes > hermes.nar
$> nix hash-file --type sha256 ./hermes.nar
warning: 'hash-file' is a deprecated alias for 'hash file'
sha256-Ug/OjuuCulhd79RCFDJLSMSPjcyvY1+XY6vwEFjaEPQ=

It appears we need to be careful to run nix flake lock --update-input <package-name> when there is a result directory in the dir below. I am going to looking into using this https://github.com/hercules-ci/gitignore.nix to ignore the results dirs when computing the nar hash.

JonathanLorimer · 2021-08-06T20:18:49Z

Confirmation that this issue appears to be resolved in CI https://github.com/JonathanLorimer/cosmos.nix/runs/3265578628?check_suite_focus=true

JonathanLorimer · 2021-08-07T02:54:46Z

Okay, I also realized I don't need the sub directories to be flakes, they can just be regular derivations.

JonathanLorimer · 2021-08-12T13:44:51Z

@soareschen I think this is ready for re-review whenever you have a moment. Thanks for taking so much time to look through this!

soareschen

LGTM. Thanks and sorry for the delay in reviewing!

hermes is building and running

4f5400f

JonathanLorimer requested a review from soareschen July 21, 2021 16:32

JonathanLorimer changed the title ~~Hermes is building and running~~ Package Hermes Jul 21, 2021

JonathanLorimer added 2 commits July 21, 2021 13:44

update description

8b35854

switch hermes to a flake

d394bb7

use hack for IFD

6c380a2

rename dir

6f62356

soareschen reviewed Jul 23, 2021

View reviewed changes

PR comments

feae1a9

JonathanLorimer requested a review from soareschen July 23, 2021 13:00

Fix README formatting

b3d8293

Fix NAR hash for hermes in top level flake

704e6c0

switch to regular derivation

08f4f4d

JonathanLorimer added 2 commits August 6, 2021 23:27

fix README alignment

054574e

minor fixes

e00c454

JonathanLorimer mentioned this pull request Aug 9, 2021

Package Gaia #8

Merged

soareschen approved these changes Aug 16, 2021

View reviewed changes

soareschen merged commit 38555f0 into informalsystems:master Aug 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package Hermes #5

Package Hermes #5

JonathanLorimer commented Jul 21, 2021 •

edited

Loading

JonathanLorimer commented Jul 21, 2021

JonathanLorimer commented Jul 21, 2021

JonathanLorimer commented Jul 21, 2021

JonathanLorimer commented Jul 21, 2021

soareschen Jul 23, 2021

soareschen Jul 23, 2021

JonathanLorimer Jul 23, 2021

soareschen Jul 23, 2021

JonathanLorimer Jul 23, 2021

soareschen left a comment

soareschen Jul 23, 2021

JonathanLorimer Jul 23, 2021

soareschen Jul 23, 2021

soareschen Jul 23, 2021

JonathanLorimer Jul 23, 2021

JonathanLorimer commented Jul 28, 2021

soareschen commented Aug 3, 2021

JonathanLorimer commented Aug 6, 2021

JonathanLorimer commented Aug 6, 2021

JonathanLorimer commented Aug 6, 2021

JonathanLorimer commented Aug 7, 2021

JonathanLorimer commented Aug 12, 2021

soareschen left a comment

Package Hermes #5

Package Hermes #5

Conversation

JonathanLorimer commented Jul 21, 2021 • edited Loading

JonathanLorimer commented Jul 21, 2021

JonathanLorimer commented Jul 21, 2021

JonathanLorimer commented Jul 21, 2021

JonathanLorimer commented Jul 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soareschen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonathanLorimer commented Jul 28, 2021

soareschen commented Aug 3, 2021

JonathanLorimer commented Aug 6, 2021

JonathanLorimer commented Aug 6, 2021

JonathanLorimer commented Aug 6, 2021

JonathanLorimer commented Aug 7, 2021

JonathanLorimer commented Aug 12, 2021

soareschen left a comment

Choose a reason for hiding this comment

JonathanLorimer commented Jul 21, 2021 •

edited

Loading