Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace /dev/urandom usage on OpenBSD #316

Closed
briansmith opened this issue Oct 17, 2016 · 22 comments
Closed

Replace /dev/urandom usage on OpenBSD #316

briansmith opened this issue Oct 17, 2016 · 22 comments

Comments

@briansmith
Copy link
Owner

See #313 (comment), where @tatsuya6502 wrote:

If you are interested in better BSD support, one obvious next step would be to replace the implementation of ring::rand with one that uses arc4random or whatever is best on BSD, instead of using reads from /dev/urandom.

OK. Let me work on that. I did a bit of research and arc4random seems the best on BSD by the following reasons:

It is widely available on BSD systems.
It produces high quality data.
It is non blocking.
http://man.openbsd.org/OpenBSD-6.0/arc4random.3

This family of functions provides higher quality data than those described in rand(3), random(3), and rand48(3).

Use of these functions is encouraged for almost all random number consumption because the other interfaces are deficient in either quality, portability, standardization, or availability. These functions can be called in almost all coding environments, including pthreads(3) and chroot(2).

That sounds good to me. The main questions I have are:

  • You say it is non-blocking. I think this is because *BSD blocks userspace programs from running until the kernel entropy pool has been initialized, right? Or, does it potentially block in the situation where the kernel entropy pool isn't initialized? Or some other behavior?
  • In your description of the functions you use "almost" twice. In what situations would using that function be inappropriate?
@briansmith briansmith mentioned this issue Oct 17, 2016
Merged
@oherrala
Copy link

There's two sets of Theo de Raadt's slides from 2014 about arc4random():

You say it is non-blocking. I think this is because *BSD blocks userspace programs from running until the kernel entropy pool has been initialized, right? Or, does it potentially block in the situation where the kernel entropy pool isn't initialized? Or some other behavior?

Kernel entropy pool is seeded by bootloader:

https://www.openbsd.org/papers/hackfest2014-arc4random/mgp00019.html

and arc4random() is used by kernel and libc (things like malloc and resolver):

I assume (since I can't find source) that kernel and libc have good entropy when userland programs can be executed so no blocking should happen.

In your description of the functions you use "almost" twice. In what situations would using that function be inappropriate?

Using word almost is direct copy from http://man.openbsd.org/arc4random.3 and the man page doesn't explain when it's inappropriate. But investigating further...

From:

https://www.openbsd.org/papers/hackfest2014-arc4random/mgp00015.html

it seems it should be always safe to call arc4random() in userland. Some exceptions only in kernel space. Also from arc4random(3):

These functions are always successful, and no return value is reserved to indicate an error.

@briansmith
Copy link
Owner Author

OK, I'm convinced arc4random is the right thing to do on OpenBSD. Also, it's what libsodium uses on OpenBSD.

Another question I have is whether the situation is different for other BSDs, in particular FreeBSD.

@tatsuya6502
Copy link
Contributor

@oherrala Thank you very much. Your research really helped me. On other day, I searched/read through man pages and blog articles on the Internet and got impression that arc4random will be the way to go. But I was not sure if I can trust those articles without seeing concrete evidences. You found all evidences, that is really great.

@briansmith

Another question I have is whether the situation is different for other BSDs, in particular FreeBSD.

I think we need to check the implementation of arc4random on every BSD systems. They share the same origin but now their kernels and base systems are different. I found man pages for arc4random in all BSD systems that I checked (OpenBSD, FreeBSD, NetBSD, and macOS), but those man pages do not tell about the implementation details.

I think we need to check at least the followings:

  • Behavior when there is not enough entropy.
  • Base algorithm. The original arc4random was based on RC4, which is now considered not secure. OpenBSD switched it to a better stream cipher ChaCha20.

As for the latter, I found this patch; FreeBSD's arc4random seems to switch to ChaCha20 in 2013.

I will continue my research, but any help from anybody would be appreciated.

@briansmith
Copy link
Owner Author

@tatsuya6502 It is also OK to write the patch and get it working on OpenBSD, and then expand its scope to other BSDs as we verify that arc4random is as safe to use as OpenBSD's. Maybe experts in other BSDs will also comment here in the interim.

tatsuya6502 added a commit to tatsuya6502/ring that referenced this issue Oct 23, 2016
@tatsuya6502
Issue briansmith#316 - Replace /dev/urandom usage on *BSD
321d710 
Now OpenBSD and FreeBSD utilizes `arc4rand_buf()` function in their
C stdlib.

- Implement GFp_sysrand_chunk() in sysrand.c for OpenBSD and FreeBSD.
- On OpenBSD and FreeBSD, mk/ring.mk now uses `-D_BSD_SOURCE` in
  `CPPFLAGS` instead of `-D_XOPEN_SOURCE=700`.

TODOs:
- Update docs.
- Remove debug print.
- Ensure FreeBSD's arc4rand_buf() is as safe as OpenBSD's.

I agree to license my contributions to each file under the terms given
at the top of each file I changed.
@tatsuya6502 tatsuya6502 mentioned this issue Oct 23, 2016
Closed
3 tasks
@tatsuya6502
Copy link
Contributor

I have got OpenBSD 6.0 (amd64) installed on my machine and built Rust 1.12.1 and Cargo from source codes. I started to make changes and opened #319 for early code review.

@tatsuya6502
Copy link
Contributor

tatsuya6502 commented Nov 1, 2016 •
edited
Loading

#316 (comment)

I think we need to check at least the followings:

  • Behavior when there is not enough entropy.
  • Base algorithm. The original arc4random was based on RC4, which is now considered not secure. OpenBSD switched it to a better stream cipher ChaCha20.

I checked FreeBSD's arc4random and /dev/(u)random implementations. My conclusion is not to use arc4random on FreeBSD and keep using /dev/urandom.

Here are some details:

arc4random

#316 (comment)

As for the latter, I found this patch; FreeBSD's arc4random seems to switch to ChaCha20 in 2013.

  • FreeBSD kern/182610: arc4random(): replace RC4 with ChaCha20, follow OpenBSD

I found above was not true. In the latest production FreeBSD release (11.0-RELEASE), arc4random is still using RC4. The source code comments say it is on RC4. We should avoid RC4 because it is known insecure.

EDIT: I think this mail discussion supports my finding.

/dev/urandom

/dev/urandom is a symbolic link to /dev/random.

/dev/random

/dev/random employs safe algorithms; Yarrow in FreeBSD 10.x-RELEASE and Fortuna in FreeBSD 11.0-RELEASE. Both algorithms are considered safe today and Fortuna is an enhanced version of Yarrow.

/dev/random will block until it is seeded by the boot loader. Once it is seeded, it will never block again.

KERN_ARND sysctl

The above man page for /dev/random also mentions KERN_ARND sysctl. I found rand crate is using it for OsRng on FreeBSD because it can be used even when /dev/(u)random is not available (chroot).

But I could not figure out if this is the same thing to reading from /dev/random. KERN_ARND sysctl calls read_random() in kernel, and this man page does not say it is the same to reading from /dev/random.

The read_random() function is used to return entropy directly from the entropy device if it has been loaded. ...
...
The read_random_uio() function behaves identically to read(2) on /dev/random. ...

So the latter is the same, but KERN_ARND uses the earlier.

I think we should continue using /dev/urandom rather than switching to KERN_ARND sysctl.

@tatsuya6502
Copy link
Contributor

I think this article gives a nice overview of the sources of entropy in different platforms. (See "General Requirements for Inclusion" section.)

Windows:

  • RtlGenRandom

Linux:

  • getrandom (if available)
    • This does the correct thing: It blocks until seeded, and then never again.
  • /dev/urandom (older Linux kernels)
    • For software that runs during the Linux boot, poll /dev/random until it's available. This means /dev/urandom has been seeded and you can safely read from /dev/urandom for all your cryptographic purposes. Don't read from /dev/random.

OpenBSD:

  • getentropy()
  • arc4random_buf() with ChaCha20 (not RC4)

Other Unix-like (including OS X):

  • /dev/urandom

The article also explains how to get secure randomness in Rust (rand, libsodium and ring crates).

Also, this Wikipedia article about Yarrow algorithm says:

Yarrow is incorporated in iOS and Mac OS X for their /dev/random devices, as did FreeBSD in the past.

So it seems good to stay with /dev/(u)random on iOS and macOS too.

@briansmith briansmith changed the title Replace /dev/urandom usage on *BSD Replace /dev/urandom usage on OpenBSD Nov 1, 2016
@briansmith briansmith mentioned this issue Nov 1, 2016
Closed
@briansmith
Copy link
Owner Author

Thanks for doing all this research. I think we should split the FreeBSD work off into its own issue, which I've already created: #326. Now this issue can be just about OpenBSD. Then we can simplify the PR to just change the behavior on BSD. WDYT?

So it seems good to stay with /dev/(u)random on iOS and macOS too.

These platforms already have their own issue: #149.

@tatsuya6502
Copy link
Contributor

tatsuya6502 commented Nov 3, 2016 •
edited
Loading

Thanks for creating #326. Also I was not aware of #149, so thank you for the pointer.

Now this issue can be just about OpenBSD. Then we can simplify the PR to just change the behavior on BSD. WDYT?

I like the idea and will simplify the PR. I first thought entire *BSD work would be just to use arc4random for all of them, but things are much more complicated.

Sorry for making very slow progress. I am still willing to work on the PR because I am learning a lot. OpenBSD is a new platform to me to do some coding, and also I have to do all those things for my job and family, so it has been hard to find some spare time. I believe I just need a few more days to finish the OpenBSD PR.

@briansmith
Copy link
Owner Author

I like the idea and will simplify the PR. I first thought entire *BSD work would be just to use arc4random for all of them, but things are much more complicated.

Thanks sounds great to me.

Sorry for making very slow progress. I am still willing to work on the PR because I am learning a lot. OpenBSD is a new platform to me to do some coding, and also I have to do all those things for my job and family, so it has been hard to find some spare time. I believe I just need a few more days to finish the OpenBSD PR.

No worries! Family first! Let me know if there's anything I can do to help (that doesn't require me to set up an OpenBSD box).

@briansmith
Copy link
Owner Author

Here are some things that should be done, based on the similar change made for macOS/iOS:

  • Update the documentation, following the example of caaa3f7. The documentation is sorted in alphabetical order.
  • Stop using on lazy_static on OpenBSD, following the example of 4410207.
  • Stop declaring a dependency on lazy_static in Cargo.toml for OpenBSD, following the example of: b88b633.

@tatsuya6502
Copy link
Contributor

Here are some things that should be done, based on the similar change made for macOS/iOS:

Thank you for the list! I will check the examples and update my code.

@briansmith briansmith mentioned this issue Feb 6, 2019
Closed
@oherrala oherrala mentioned this issue Feb 24, 2019
Closed
@josephlr josephlr mentioned this issue Jun 23, 2019
Open
@briansmith
Copy link
Owner Author

PR #874 restores the /dev/urandom-based implementation for OpenBSD.

@briansmith
Copy link
Owner Author

If you are somebody that uses Rust on OpenBSD, could you please explain how you build Rust programs? How do you get libstd for OpenBSD? Are you using xargo?

@oherrala
Copy link

@briansmith OpenBSD has packaged Rust and Cargo in Ports. @semarie is the maintainer.

https://github.com/openbsd/ports/tree/master/lang/rust

So, the workflow is not much different from any other supported platform for Rust:

  • pkg_add rust
  • cargo build / run / install

@briansmith
Copy link
Owner Author

Thanks @oherrala

So, the workflow is not much different from any other supported platform for Rust:

For every other target, including Windows, I can build on a Linux host (including Microsoft Linux on Windows) or macOS host. What's the easiest way to build targeting OpenBSD from Linux or macOS?

@oherrala
Copy link

What's the easiest way to build targeting OpenBSD from Linux or macOS?

AFAIK there's no cross compilation to target OpenBSD.

@semarie
Copy link

semarie commented Aug 3, 2019

@briansmith as nobody worked on crosscompilation from Linux to OpenBSD (complete C & C++ toolchain with linker), there is no simple solution currently to targeting OpenBSD from Linux (or macos).

it could be tried with clang+lld as OpenBSD uses them for x86_64. but I dunno if there is still some parts in linker that aren't upstreamed.

@briansmith
Copy link
Owner Author

Thanks for the info. It would be very convenient if it were possible to build and link for OpenBSD targets from a Linux (or macOS) host. Even though we have to run tests on OpenBSD itself, that can easily be done in a small and limited VM. But, doing the build itself would require a large (in terms of CPU and memory capability, as well as installed binaries) VM. I would love to hear from anybody trying to upstream changes needed to the Rust toolchain (and LLVM itself?) to make this happen.

@cemeyer
Copy link

cemeyer commented Nov 24, 2020 •
edited
Loading

For what it's worth, FreeBSD's userspace arc4random(3) is based on Chacha20 in FreeBSD 12.x and newer (but not 11.x and older). 11.x is still supported, as far as I know, but might be end-of-line very soon with the upcoming 13.0 release.

The 2013 reference is to an email of a bug / proposed patch that was incomplete until 2017/2018. Here's a better link to the same bug, with more context: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=182610

@briansmith
Copy link
Owner Author

See the draft PR #1531 where I propose we delegate all this to the getrandom crate.

@briansmith
Copy link
Owner Author

PR #1531 was merged, which switched to using getrandom on OpenBSD, so I'm closing this issue.

Also, FreeBSD users, please check out PR #1542, which further improves support for OpenBSD.

@fabrice102 fabrice102 mentioned this issue Jun 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

On OpenBSD, use arc4random_buf to implement SystemRandom::fill() oherrala/ring
5 participants
@briansmith @oherrala @cemeyer @tatsuya6502 @semarie