Failed to verify ink contracts #1

saki-osive · 2023-01-06T14:40:23Z

I followed the tutorial provided here for verifying ink smart contracts. However, I'm facing an error:

The script in the ink-verifier-image repo under scripts/verify-contract is failing at line number 26:

..
mkdir -p $TARGET
..

Is there a step missing in the tutorial?

The text was updated successfully, but these errors were encountered:

XY-Wang · 2023-01-09T10:54:26Z

Hi,

it seems like the container does not have the permission to create the directory build/package.

This is how it works:

Ink verifier server runs in its own process, with the user that executes the command.
This process creates a processing directory in the host file system that will have the default permissions according to the umask of your system. By default, the processing directory is located at ./.tmp/processing/<network_id>/<code_hash>.
When a verification process is triggered it will run a docker container mapping the processing directory to the /build directory inside the container.
If the container is not running in rootless mode or the OS is not SELinux it should work with the default configuration, since the container uses uid 0.

Note that we tested it out in rootless docker and with podman, is a more secure setup but it needs extra configuration.

Could you please provide the output to the command docker info?

saki-osive · 2023-01-09T12:41:06Z

@XY-Wang @mfornos Sure.

Here is the output of the docker info command:

arcturus@arcturus:~$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 10
  Running: 1
  Paused: 0
  Stopped: 9
 Images: 62
 Server Version: 20.10.16
 Storage Driver: zfs
  Zpool: rpool
  Zpool Health: ONLINE
  Parent Dataset: rpool/var/lib/docker
  Space Used By Parent: 8162181120
  Space Available: 389698441216
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.0-28-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 62.71GiB
 Name: arcturus
 ID: NS34:HLO5:ULJB:EG2X:XRCG:YC3R:OHMX:C44K:TXNM:BO6N:UH37:2MQ6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Does it provide any help?
Thanks.

XY-Wang · 2023-01-09T15:53:36Z

Hmm... did you happen to be able to build a verifiable package with the command line tool according to this part of the tutorial? If you haven't, could you please try it to see if you get similar errors?

Could you also provide the output of an ls -la command in the [path]/ink-verifier-server/.tmp/ directory?

saki-osive · 2023-01-09T18:24:37Z

@XY-Wang Thanks for the reply.

Yes, the command line tool named build-verifiable-ink was built successfully and is available as a cargo binary:

The verifiable package was successfully built using:

..

Generate the verifiable source code package

build-verifiable-ink -t develop .
..

Here is the output of the [path]/ink-verifier-server/.tmp directory:

XY-Wang · 2023-01-10T09:00:39Z

Could you please try to remove the staging and processing folders in .tmp/ and restart the server? Do you still run into the same problem?

Are you able to do a docker container inspect on the verification container while it's running?

XY-Wang · 2023-01-17T08:53:40Z

Hi @saki-osive, did you manage to solve the issue?

saki-osive · 2023-01-23T16:51:47Z

Could you please try to remove the staging and processing folders in .tmp/ and restart the server? Do you still run into the same problem?

Are you able to do a docker container inspect on the verification container while it's running?

@XY-Wang I tried these steps but still, I'm getting the same error.

XY-Wang · 2023-01-23T18:16:40Z

@saki-osive could you paste the output of the following commands?

docker run -it --privileged --entrypoint id ink-verifier:develop
docker run -it --entrypoint ls -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop -la .

saki-osive · 2023-01-23T19:44:50Z

Sure @XY-Wang

Running

arcturus@pop-os:~/patch/ink_verification/ink-verifier-image$ docker run -it --privileged --entrypoint id ink-verifier:develop

outputs

uid=0(root) gid=0(root) groups=0(root)

Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --entrypoint ls -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -la .

outputs

total 12
drwxr-xr-x  3 root root 4096 Jan 23 16:36 .
drwxr-xr-x 23 root root 4096 Jan 23 19:40 ..
drwxr-xr-x  3 root root 4096 Jan 23 16:47 ink

mfornos · 2023-01-23T20:25:32Z

Just to confirm, you didn't modified the apparmor default docker profile and the create directory is not being forbidden by apparmor (`demsg | grep apparmor | grep docker` to check)?

…

On Mon, 23 Jan 2023, 20:45 saki-osive, ***@***.***> wrote: Sure @XY-Wang <https://github.com/XY-Wang> 1. Running ***@***.***:~/patch/ink_verification/ink-verifier-image$ docker run -it --privileged --entrypoint id ink-verifier:develop outputs uid=0(root) gid=0(root) groups=0(root) 1. Running ***@***.***:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --entrypoint ls -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -la . outputs total 12 drwxr-xr-x 3 root root 4096 Jan 23 16:36 . drwxr-xr-x 23 root root 4096 Jan 23 19:40 .. drwxr-xr-x 3 root root 4096 Jan 23 16:47 ink — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFCCICOBW7MGCREBTY3VBDWT3NTZANCNFSM6AAAAAATTESXHU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

saki-osive · 2023-01-23T20:39:00Z

@mfornos
Ummm.. Although a little unclear, I assume you're asking if I modified the docker profile.

No, I didn't.

mfornos · 2023-01-23T21:14:59Z

So, if you run docker run -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop the verification gives you a different error?

saki-osive · 2023-01-24T01:24:11Z

So, if you run docker run -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop the verification gives you a different error?

Hi @mfornos , running this gives me the output:
No archive found

mfornos · 2023-01-24T08:53:13Z

We will try to figure out if the problem is related with the default parameters that ink verifier server uses to run the container.

Please, try the following commands:

docker run -it --entrypoint mkdir -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop -v a
docker run -it --security-opt=no-new-privileges --cap-drop all --entrypoint mkdir -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop -v b

saki-osive · 2023-01-24T13:09:26Z

@mfornos Sure,

Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --entrypoint mkdir -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -v a

outputs

mkdir: created directory 'a'

Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --security-opt=no-new-privileges --cap-drop all --entrypoint mkdir -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -v b

outputs

mkdir: created directory 'b'

mfornos · 2023-01-24T15:06:29Z

The permission problem is not reproducible with the docker run, let's check if it is related with node.js.

Please, save the attached test_docker_run.zip script, unzip and execute it:
node test_docker_run.js /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target/test

saki-osive · 2023-01-24T18:45:33Z

Sure @mfornos,

Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ node test_docker_run.js /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target/test

outputs

Spawn Docker params: [
  'run',
  '--rm',
  '--security-opt=no-new-privileges',
  '--cap-drop',
  'all',
  '-v',
  '/home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target/test:/build',
  '--entrypoint',
  'mkdir',
  'ink-verifier:develop',
  '-v',
  'xyz'
] 
---

User Info: {
  uid: 1000,
  gid: 1000,
  username: 'arcturus',
  homedir: '/home/arcturus',
  shell: '/bin/bash'
} 
---

mkdir: created directory 'xyz'
248170 exit 0

XY-Wang · 2023-01-25T14:56:24Z

Hmmm this is really strange since the script mimics the process of ink-verifier-server
i.e. start a node process which spawns a docker container which tries to create a directory
We expected that you'd run into the same error but clearly that's not the case...

From the test docker runs it seems like your docker is running with root access so in theory there shouldn't be any permission issues.

The only thing we can see here, for the moment, is that in the earlier screenshots you were running commands from arcturus@arcturus and in the most recent successful runs it has changed to arcturus@pop-os. Are you running in a different host or has your machine set up been changed? If you run the server from arcturus@pop-os do you still get the same permission issue?

saki-osive · 2023-01-25T14:58:48Z

Hi @XY-Wang Thanks for the reply.

The set-up didn't seem to work on either of the machines, pop-os or arcturus is just the hostname, which can be changed as well.

mfornos · 2023-01-25T15:49:06Z

🤔 As @XY-Wang said the server does the same as the troubleshooting script we provided.

There is no apparent logical reason for it to fail creating the directory and we cannot reproduce the permission issue, not in our development machines, not in the servers (running other OS), not even in your own host with the provided script.

Do you have an AppArmor profile for the npm command?
You could check your profiles in /etc/apparmor.d/ and see the ones being applied executing sudo aa-status

saki-osive · 2023-01-25T18:33:24Z

Sure @mfornos,

Running

arcturus@pop-os:/etc/apparmor.d$ ls /etc/apparmor.d/

outputs

abi             local            tunables         usr.lib.libreoffice.program.oosplash     usr.sbin.chronyd
abstractions    lsb_release      usr.bin.evince   usr.lib.libreoffice.program.senddoc      usr.sbin.cups-browsed
disable         nvidia_modprobe  usr.bin.man      usr.lib.libreoffice.program.soffice.bin  usr.sbin.cupsd
force-complain  sbin.dhclient    usr.bin.tcpdump  usr.lib.libreoffice.program.xpdfimport   usr.sbin.rsyslogd

Running

arcturus@pop-os:/etc/apparmor.d$ sudo aa-status

outputs

apparmor module is loaded.
27 profiles are loaded.
25 profiles are in enforce mode.
   /usr/bin/evince
   /usr/bin/evince-previewer
   /usr/bin/evince-previewer//sanitized_helper
   /usr/bin/evince-thumbnailer
   /usr/bin/evince//sanitized_helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/cups/backend/cups-pdf
   /usr/sbin/chronyd
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /{,usr/}sbin/dhclient
   docker-default
   libreoffice-senddoc
   libreoffice-soffice//gpg
   libreoffice-xpdfimport
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   tcpdump
2 profiles are in complain mode.
   libreoffice-oosplash
   libreoffice-soffice
0 profiles are in kill mode.
0 profiles are in unconfined mode.
58 processes have profiles defined.
58 processes are in enforce mode.
   /usr/sbin/chronyd (1240) 
   /usr/sbin/chronyd (1244) 
   /usr/sbin/cups-browsed (237056) 
   /usr/sbin/cupsd (237027) 
   /usr/local/bin/node (25663) docker-default
   /usr/local/bin/node (26227) docker-default
   /usr/lib/postgresql/15/bin/postgres (26543) docker-default
   /usr/local/bin/substrate-contracts-node (26550) docker-default
   /usr/sbin/nginx (26748) docker-default
   /usr/lib/postgresql/15/bin/postgres (26934) docker-default
   /usr/local/bin/node (27317) docker-default
   /usr/lib/postgresql/15/bin/postgres (27330) docker-default
   /usr/lib/postgresql/15/bin/postgres (27331) docker-default
   /usr/lib/postgresql/15/bin/postgres (27451) docker-default
   /usr/lib/postgresql/15/bin/postgres (27452) docker-default
   /usr/lib/postgresql/15/bin/postgres (27453) docker-default
   /usr/sbin/nginx (27612) docker-default
   /usr/sbin/nginx (27613) docker-default
   /usr/sbin/nginx (27614) docker-default
   /usr/sbin/nginx (27615) docker-default
   /usr/sbin/nginx (27617) docker-default
   /usr/sbin/nginx (27618) docker-default
   /usr/sbin/nginx (27619) docker-default
   /usr/sbin/nginx (27620) docker-default
   /usr/sbin/nginx (27621) docker-default
   /usr/sbin/nginx (27622) docker-default
   /usr/sbin/nginx (27623) docker-default
   /usr/sbin/nginx (27624) docker-default
   /usr/sbin/nginx (27625) docker-default
   /usr/sbin/nginx (27626) docker-default
   /usr/sbin/nginx (27627) docker-default
   /usr/sbin/nginx (27628) docker-default
   /usr/sbin/nginx (27629) docker-default
   /usr/sbin/nginx (27630) docker-default
   /usr/sbin/nginx (27631) docker-default
   /usr/sbin/nginx (27632) docker-default
   /usr/sbin/nginx (27633) docker-default
   /usr/sbin/nginx (27634) docker-default
   /usr/sbin/nginx (27635) docker-default
   /usr/sbin/nginx (27636) docker-default
   /usr/sbin/nginx (27637) docker-default
   /usr/sbin/nginx (27638) docker-default
   /usr/sbin/nginx (27639) docker-default
   /usr/sbin/nginx (27640) docker-default
   /usr/sbin/nginx (27641) docker-default
   /usr/sbin/nginx (27642) docker-default
   /usr/sbin/nginx (27643) docker-default
   /usr/sbin/nginx (27644) docker-default
   /usr/lib/postgresql/15/bin/postgres (27742) docker-default
   /usr/lib/postgresql/15/bin/postgres (27743) docker-default
   /usr/lib/postgresql/15/bin/postgres (27745) docker-default
   /usr/lib/postgresql/15/bin/postgres (27746) docker-default
   /usr/lib/postgresql/15/bin/postgres (27747) docker-default
   /substrate-gateway/substrate-gateway (27749) docker-default
   /usr/local/bin/node (27866) docker-default
   /usr/local/bin/node (28806) docker-default
   /usr/lib/postgresql/15/bin/postgres (28928) docker-default
   /usr/lib/postgresql/15/bin/postgres (688933) docker-default
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.

saki-osive · 2023-01-27T00:16:25Z

🤔 As @XY-Wang said the server does the same as the troubleshooting script we provided.

There is no apparent logical reason for it to fail creating the directory and we cannot reproduce the permission issue, not in our development machines, not in the servers (running other OS), not even in your own host with the provided script.

Do you have an AppArmor profile for the npm command? You could check your profiles in /etc/apparmor.d/ and see the ones being applied executing sudo aa-status

Well I have tried it 10+ times on 4+ systems with different operating systems. I guess there's some step missing from the tutorial.

@XY-Wang @mfornos

mfornos · 2023-02-08T17:28:25Z

Hi @saki-osive

Finally we reproduced the issue! 🐸

It was caused by the "--cap-drop all" security flag in the docker run command. Nevertheless you will still have other problems related to running the container mapped to the root user in the host.

So, to avoid more issues, we have decided to update the tutorial with a prerequisite configuration of the docker daemon (i.e. user namespace remapping to the host user id).

Check it here, please: https://github.com/web3labs/ink-verifier-server/blob/main/docs/TUTORIAL.md#docker-user-namespace-remapping

saki-osive · 2023-02-09T06:31:49Z

Hi @mfornos , thank you for the reply.

I re-tested the updated doc.

However, I'm facing this issue:

Any idea how this can be fixed?

@XY-Wang

mfornos · 2023-02-09T08:00:43Z

We have added many changes in the system, so you need to re-deploy everything from scratch.

…

On Thu, 9 Feb 2023, 07:32 saki-osive, ***@***.***> wrote: Hi @mfornos <https://github.com/mfornos> , thank you for the reply. I re-tested the updated doc. However, I'm facing this issue: [image: image] <https://user-images.githubusercontent.com/62865682/217735498-81f46622-fe76-4ba4-95e3-4b222ab94480.png> Any idea how this can be fixed? @XY-Wang <https://github.com/XY-Wang> — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFCCIDBSLQWIYYVJVJ7WILWWSFODANCNFSM6AAAAAATTESXHU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

XY-Wang · 2023-02-09T08:50:03Z

Hi @saki-osive ,

Please do a git pull on all the repos since we've made many changes 😃 Also, since you've restarted docker all your existing docker images would have been wiped, so you'll need to run docker build . -t ink-verifier:develop again.

We've amended in the tutorial the command to run ink-verifier-server to include setting the server host to 0.0.0.0 so that it can be accessed from the docker container. Check it out here.

Please also note that if you're running the docker compose files in epirus-substrate/local-testnet/ separately (i.e. docker-compose -f docker-compose.{SERVICE}.yml up -d) instead of using the run-all.sh script, you'll need to do a docker-compose -f docker-compose.{SERVICE}.yml pull to get the latest image. If you run the run-all.sh script the pull is already taken care of.

Hope that helps!

saki-osive · 2023-02-09T11:57:39Z

Hi @mfornos @XY-Wang , thank you for the reply.

I removed all directories, docker images, volumes and containers. After that I cloned all repos and started everything from scratch.

Steps I followed:

Ran ./run-all.sh under epirus-substrate/local-testnet : RAN SUCCESSFULLY
npm install and SERVER_HOST=0.0.0.0 npm start under ink-verifier-server: RAN SUCCESSFULLY
Ran docker build . -t ink-verifier:develop under ink-verifier-image: RAN SUCCESSFULLY
Ran cargo install --path . under cli/: RAN SUCCESSFULLY and built the binary
Ran build-verifiable-ink -t develop . under dev-contracts-substrate/example-contracts/flipper/: RAN SUCCESSFULLY and outputed the contract files
Then I uploaded the contract on this UI.
The contract was uploaded successfully but it kept loading during calling it.
Finally, I was able to make 3-4 calls on the flipper and bool functions to reverse the true and false values after reloading the UI 2-3 times.

After this, I went to localhost:3000, but it again showed me like this:

The error message is this:

I have tested this 3-4 times on different systems. I believe something is broken/not documented?

XY-Wang · 2023-02-09T12:19:33Z

Hmm... could you please paste the logs of the ./run-all.sh when this error happens?

saki-osive · 2023-02-09T13:39:24Z

Hi @XY-Wang, I stopped all the running containers and invoked the command again and it worked. But still doesn't work on the first attempt, maybe it needs reloading?

XY-Wang · 2023-02-09T14:21:05Z

That's great! The issue with the first run is probably that your browser had the old version cached 😄

saki-osive · 2023-02-09T15:39:10Z

That's great! The issue with the first run is probably that your browser had the old version cached smile

Can't confirm it's a browser cache issue as I tried in Incognito mode and another browser too.

mfornos added question Further information is requested setup Setup related issues labels Jan 9, 2023

mfornos added documentation Improvements or additions to documentation and removed question Further information is requested labels Feb 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to verify ink contracts #1

Failed to verify ink contracts #1

saki-osive commented Jan 6, 2023 •

edited

Loading

XY-Wang commented Jan 9, 2023

saki-osive commented Jan 9, 2023

XY-Wang commented Jan 9, 2023

saki-osive commented Jan 9, 2023 •

edited

Loading

XY-Wang commented Jan 10, 2023

XY-Wang commented Jan 17, 2023

saki-osive commented Jan 23, 2023

XY-Wang commented Jan 23, 2023

saki-osive commented Jan 23, 2023

mfornos commented Jan 23, 2023 via email

saki-osive commented Jan 23, 2023

mfornos commented Jan 23, 2023 •

edited

Loading

saki-osive commented Jan 24, 2023

mfornos commented Jan 24, 2023

saki-osive commented Jan 24, 2023

mfornos commented Jan 24, 2023

saki-osive commented Jan 24, 2023

XY-Wang commented Jan 25, 2023

saki-osive commented Jan 25, 2023

mfornos commented Jan 25, 2023

saki-osive commented Jan 25, 2023

saki-osive commented Jan 27, 2023

mfornos commented Feb 8, 2023

saki-osive commented Feb 9, 2023

mfornos commented Feb 9, 2023 via email

XY-Wang commented Feb 9, 2023

saki-osive commented Feb 9, 2023

XY-Wang commented Feb 9, 2023

saki-osive commented Feb 9, 2023

XY-Wang commented Feb 9, 2023

saki-osive commented Feb 9, 2023

Failed to verify ink contracts #1

Failed to verify ink contracts #1

Comments

saki-osive commented Jan 6, 2023 • edited Loading

XY-Wang commented Jan 9, 2023

saki-osive commented Jan 9, 2023

XY-Wang commented Jan 9, 2023

saki-osive commented Jan 9, 2023 • edited Loading

XY-Wang commented Jan 10, 2023

XY-Wang commented Jan 17, 2023

saki-osive commented Jan 23, 2023

XY-Wang commented Jan 23, 2023

saki-osive commented Jan 23, 2023

mfornos commented Jan 23, 2023 via email

saki-osive commented Jan 23, 2023

mfornos commented Jan 23, 2023 • edited Loading

saki-osive commented Jan 24, 2023

mfornos commented Jan 24, 2023

saki-osive commented Jan 24, 2023

mfornos commented Jan 24, 2023

saki-osive commented Jan 24, 2023

XY-Wang commented Jan 25, 2023

saki-osive commented Jan 25, 2023

mfornos commented Jan 25, 2023

saki-osive commented Jan 25, 2023

saki-osive commented Jan 27, 2023

mfornos commented Feb 8, 2023

saki-osive commented Feb 9, 2023

mfornos commented Feb 9, 2023 via email

XY-Wang commented Feb 9, 2023

saki-osive commented Feb 9, 2023

XY-Wang commented Feb 9, 2023

saki-osive commented Feb 9, 2023

XY-Wang commented Feb 9, 2023

saki-osive commented Feb 9, 2023

saki-osive commented Jan 6, 2023 •

edited

Loading

saki-osive commented Jan 9, 2023 •

edited

Loading

mfornos commented Jan 23, 2023 •

edited

Loading