sm_expected_hash.h is missing

[benlaurie]

I get as far asmake trusted_client.riscv and it fails with:

trusted_client/trusted_client.cpp:7:10: fatal error: sm_expected_hash.h: No such file or directory
 #include "sm_expected_hash.h"
          ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:47: recipe for target 'trusted_client.riscv' failed

[dkohlbre]

We're currently in the middle of trying to figure out the best way to handle expected hash management. For enclave applications this isn't too bad, so we ship the current expected hash along with the newest enclave app version.

Unfortunately for the security monitor, the hash is different depending on the target platform type (currently, between qemu and the hifive test board) since they use different boot processes.

I'll put together a temporary set of instructions for generating/using a valid hash, and add a valid sm hash to the riscv-pk repo for now.

If you've got any ideas we'd love to hear them!

[benlaurie]

I'll think about that (haven't even seen the riscv-pk repo yet, tho), but in the meantime I worked around that (by just making a hash up, which somewhat disturbingly has not obviously broken anything), and now I get this:

Enclave said: NOT USING REAL RANDOMNESS: TEST ONLY
{"device_pubkey": "0faad4ff01178583baa588966f7c1ff32564dd17d7dc2b46cb50a84a69270b4c", "enclave": {"data": "22b18fa141a92c8d7208d5d463807d8e5fab80fa9fb143c90651611956e0fa61", "datalen": 32, "hash": "26812337903153ffe4ae7ff1260f572468d4b1fbb4ae0c883c1f56e263bfe7385a996c02a7224da6e1b6054bed9a2d04ae0995e9696901c93ff92fa793dec34d", "signature": "a8d700f9809edce0d93ee414c5b7151d4ea9809a65b391baada5129728dc78c294c54159192f38df136aaaf1e2f8c6bbc2f67b23f6278f4ce8ef55717e94010f"}, "security_monitor": {"hash": "2c60d3d52e44a4aa290b34dd19a08fbcf3270e38b3c5f1adfc0a0c2cb6dbde8ddea026b59adac9211d3e6b0fdfc5031c70a4f31c3e10e33815fb866f68615553", "pubkey": "a1e6298d34c3978ed047e6745bb00630bcf22edcc1fe08f08541bece9315eb9a", "signature": "c4d2b36bae63719f920cfb4caed14f50ad489416ebbb2a86bf2ba47a688cba110d2e1937e6c8c150bedaabc7bbe6268bde14ab307cf43b132f9e019b9f94e903"}}
Attestation report SIGNATURE is valid
Enclave said: [C] Unable to generate seesion keys, exiting

[dkohlbre]

I've pushed a working hash for the current sm, and updated the BUILDING file to indicate to copy it for now.

I think you are on an old version of the keystone-demo, that hash and error message don't match up with the current enclave. We are currently moving to doing development in a new flow, new commits to the keystone meta-repo should be showing up under the dev branch shortly. You can pull to the latest version of the submodules if the dev branch isn't up to date yet.

Bad hashes absolutely break the attestation flow (I just added a feature to the demo only for ignoring that during testing since generating the hash files is manual right now.)

[benlaurie]

I only checked out keystone-demo a few hours ago.

Not sure what you mean about the new flow and dev branch? which one is the meta repo? are you saying I should be on dev not master? or what?

[dkohlbre]

Apologies for the confusion. You checked it out while I was in the middle of merging in feature changes we made for our hardware demo. I'm getting things cleaned up right now :)

If you checked out keystone/keystone (the meta-repo, containing mostly submodules) make sure you check out the dev branch of that.

Second, for the keystone-demo, I've removed the extraneous branches. You should be on master for that.

I realized the issue is likely that you are using the dummy client, which was supposed to get removed during a recent merge. It is no longer supported. Updated instructions on running the demo are available here momentarily: http://docs.keystone-enclave.org/en/dev/Keystone-Demo/Building.html

I'll remove the old dummy client code.

[benlaurie]

Ah, I see. I'm currently on a plane so I'm not sure how wise it would be to switch branches (given the many gigabytes of download the original one was) but I can give it a try.

[dkohlbre]

I just bumped and double checked the dev branch for keystone, and the master branch for keystone-demo.

If you were working off keystone:master rather than keystone:dev we also switched to a dynamically loaded kernel module. This shouldn't affect the demo itself but the module now needs to be loaded before the host will work. (http://docs.keystone-enclave.org/en/dev/Getting-Started/Running-Keystone-with-QEMU.html#insert-keystone-driver)

Let me know if you run into any issues with the trusted_client version.

[dkohlbre]

As a heads up, we're removing the demo documentation from the main docs page, and moving it into the keystone-demo repository. I'm also updating them based on the problems you've run into.

Thanks for sticking with us :) Should be clear and up-to-date docs available shortly for the demo.

[benlaurie]

Hmm. The switch to the dev branch has not gone well. I'm pretty good with git generally, but I find submodules confusing, I guess because I don't use them much.

What am I supposed to do after checking out the dev branch to get the submodules back in sync?

I fear I've broken my repo anyway, so that may be it for the day.

[dayeol]

Hi, Can you try git submodule sync git submodule update --init and see if there's no error? and if linux-keystone-driver is checked out. Thanks Dayeol

…

On Fri, Jan 11, 2019, 12:37 PM Ben Laurie ***@***.*** wrote: Hmm. The switch to the dev branch has not gone well. I'm pretty good with git generally, but I find submodules confusing, I guess because I don't use them much. What am I supposed to do after checking out the dev branch to get the submodules back in sync? I fear I've broken my repo anyway, so that may be it for the day. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB9sJ1ofLvfkxilYdKJBK8m7NzCWp7T2ks5vCPYCgaJpZM4Z7qWz> .

[benlaurie]

Finally had time to try this. All works until I try ./test inside qemu, and then it fails with lots of this kind of thing:

# ./test
testing stack
[Keystone SDK] src/keystone.cpp:78 : failed to create enclave - ioctl() failed: -1

[dkohlbre]

If you are running the ./test script supplied by the sdk tests, I think its not currently insmod'ing the keystone driver by default since we made the switch to a module.

Both the sdk tests and the demo should work if you do:
insmod keystone-driver.ko

I'll go add that to the test script.

[dayeol]

If you haven't compiled linux-keystone-driver, you need to compile it and copy the driver into disk image first.

cd linux-keystone-driver
make
make copy
cd ..

and as David mentioned, you need to insert the module in QEMU guest

insmod keystone-driver.ko

I know it is pretty annoying to manually run each of the commands.
I'll package the entire build process since it seems pretty stable now.

[benlaurie]

I was already doing that, so that's not it.

BTW, I have the build as a makefile already. I was waiting for it to work before I sent a PR.

[dkohlbre]

Does that happen the first time you run the ./test script immediately after booting qemu?

We've seen that problem when the module isn't loaded, or when we've had a memory leak or similar problem and the driver has run out of places to allocate enclaves in the past. So if the module is loaded and this is the first run of the boot we'll have to dig into it.

Can you run git submodule status and give us the output of that from the main keystone repository?

[benlaurie]

Yes, immediately after booting.

 74063847e613931082eba7a11b66dea289d4e35a busybear-linux (v1.0-12-g7406384)
 4f8e0ea159b8e76a33c53b72b2921d592bb93076 linux-keystone-driver (heads/master)
 1b80cbe97d2c29212398d3b74ddc54718ad32e23 riscv-gnu-toolchain (v20171231-6-g1b80cbe)
 65e929792fb9b632c20be118fa0795b26cc89fe4 riscv-linux (v4.15-60-g65e9297)
 bccebfccc052baaea188c3e0490b6843320e968f riscv-pk (archive-1.9.1-130-gbccebfc)
 3b3e2ed8a6ed38861422f199a32e58dfc5f6358c riscv-qemu (v3.0.0-rc2-83-g3b3e2ed)
 c7a0a6e7c1fb9f24c725acd1766244f80ee87487 sdk (remotes/origin/HEAD)

[dayeol]

All submodules seem to be correct.

The most likely causes of the problem are:
(1) the security monitor failed to be initialized
(2) the driver could not create /dev/keystone_enclave file.

You can check (1) by accessing the security monitor memory 0x80000000.
Due to a known bug in QEMU, you need to access 0x40000000 instead.
(keystone-enclave/keystone#25)

busybox devmem 0x40000000

If above command generates a kernel fault, it means that the security monitor is initialized, and the memory is protected by PMP.

You can check (2) by
ls /dev/keystone_enclave
after insmod.
/dev/keystone_enclave file should have been created if insmod did not fail.

Could you let us know if one of two were the cause?

Also FYI, we have a script for travis CI in keystone/scripts/travis.sh, which contains the packaged compilation commands.

[benlaurie]

# busybox devmem 0x40000000
[   73.522019] busybox[70]: unhandled signal 11 code 0x2 at 0x00000000000201f4 in busybox[10000+d4000]
[   73.525990] CPU: 0 PID: 70 Comm: busybox Tainted: G           O     4.15.0-00060-g65e9297 #2
[   73.528528] sepc: 00000000000201f4 ra : 000000000002013e sp : 0000003fff8e2ae0
[   73.531242]  gp : 00000000000e65e8 tp : 00000020001bb710 t0 : 000000000000013a
[   73.536474]  t1 : 00000020001538fe t2 : ffffffffffffffff s0 : 0000000000000020
[   73.538549]  s1 : 0000003fff8e2f69 a0 : 0000002000017000 a1 : 0000000000001000
[   73.539975]  a2 : 0000000000000001 a3 : 0000000000000001 a4 : 0000000000000003
[   73.541109]  a5 : 0000000000000020 a6 : 0000000000000000 a7 : 00000000000000de
[   73.542226]  s2 : 0000003fff8e2dc8 s3 : 0000000000000000 s4 : 0000000040000000
[   73.543192]  s5 : 0000003fff8e2f69 s6 : 00000000000e6120 s7 : 0000000000000002
[   73.544420]  s8 : 0000000000000014 s9 : 00000020001bb720 s10: 0000000000000000
[   73.545461]  s11: 0000003fffda9fa2 t3 : 000000000009e8fe t4 : 0000000000000002
[   73.546733]  t5 : 00000020000b6eec t6 : 0000000000000000
[   73.548043] sstatus: 0000000000002020 sbadaddr: 000000200014ce7a scause: 0000000000000005
Segmentation fault
# ls -l /dev/keystone_enclave 
crw-rw----    1 root     root       10,  62 Jan  1 00:00 /dev/keystone_enclave

[dayeol]

I still couldn't reproduce the issue in my end.

Could you help me debug the issue by adding perror("error"); at line 79 in file sdk/lib/host/src/keystone.cpp as shown below?

  if(ret) {
    ERROR("failed to create enclave - ioctl() failed: %d", ret);
    perror("error"); // <---- 
    return KEYSTONE_ERROR;
  }

You might need to re-compile sdk and copy binaries before launching QEMU.

cd sdk
make
make copy-tests
cd ..

[dayeol]

Also, did you rebuild riscv-linux, riscv-pk, and sdk after checkout to dev branch?

[benlaurie]

The error is "Inappropriate ioctl for device".

And yes, I rebuilt everything. Note, by "rebuilt" I mean "followed the build instructions". I don't mean that I cleaned everything and built again.

[dayeol]

Opened a new issue keystone-enclave/keystone#29.

[dayeol]

@benlaurie, if you are still getting the same error, can you try with a fresh repo,
checkout to dev branch,
and run ./build.sh ?
This will take about 3 hours, but ./fast-build.sh will take less than an hour.

fast-build.sh uses pre-built riscv toolchain instead of compiling all.

I believe that the problem is somehow caused while moving around the branches.

Also could you let me know which OS are you using?

[dayeol]

Closing the thread because it was inactive for a while.